HTTP-01 challenge can't reach local server

Client dev
1

I'm attempting to test simple Java client app with acme4j library locally with latest Pebble Docker image. The account creation part is completed successfully, but the HTTP-01 validation stage fails with various status codes, depending on default IPv4 settings for challtestsrv.

For testing, I run a simple Python HTTP server which contains only .well-known/acme-challenge/<TOKEN> file. The command for it: python -m http.server 5002 (Python Version 3.10.1). To ensure that file is accessible, I used curl localhost:5002/.well-known/acme-challenge/<REQUIRED_NAME>, which returned the correct content.

When using the default docker-compose.yml, I receive the following error during this challenge:

The key authorization file from the server did not match this challenge \"ZxZaPZD5LtpTD6TBnTysD4jOWzsQvws-sCoWUxNkeZ0.yeKaorVdAYaqeyJcRI3KzwQwTDf6plJwZMkNvTS9K-k\" != \"\"","status":403

I tried to debug it with Wireshark and found, that the request is sent to 10.30.50.3:5002:

Frame 117: 294 bytes on wire (2352 bits), 294 bytes captured (2352 bits) on interface vethf215e22, id 1
Ethernet II, Src: 02:42:0a:1e:32:02 (02:42:0a:1e:32:02), Dst: 02:42:0a:1e:32:03 (02:42:0a:1e:32:03)
Internet Protocol Version 4, Src: 10.30.50.2, Dst: 10.30.50.3
Transmission Control Protocol, Src Port: 48652, Dst Port: 5002, Seq: 1, Ack: 1, Len: 228
    Source Port: 48652
    Destination Port: 5002
    [Stream index: 3]
    [Conversation completeness: Complete, WITH_DATA (31)]
    [TCP Segment Len: 228]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 106143684
    [Next Sequence Number: 229    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 2934519440
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 502
    [Calculated window size: 64256]
    [Window size scaling factor: 128]
    Checksum: 0x794b [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
        [Time since first frame in this TCP stream: 0.000563568 seconds]
        [Time since previous frame in this TCP stream: 0.000420927 seconds]
    [SEQ/ACK analysis]
        [iRTT: 0.000142641 seconds]
        [Bytes in flight: 228]
        [Bytes sent since last PSH flag: 228]
    TCP payload (228 bytes)
Hypertext Transfer Protocol
    GET /.well-known/acme-challenge/ZxZaPZD5LtpTD6TBnTysD4jOWzsQvws-sCoWUxNkeZ0 HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET /.well-known/acme-challenge/ZxZaPZD5LtpTD6TBnTysD4jOWzsQvws-sCoWUxNkeZ0 HTTP/1.1\r\n]
        Request Method: GET
        Request URI: /.well-known/acme-challenge/ZxZaPZD5LtpTD6TBnTysD4jOWzsQvws-sCoWUxNkeZ0
        Request Version: HTTP/1.1
    Host: test.mypythonserver.gg:5002\r\n
    User-Agent: LetsEncrypt-Pebble-VA (linux; amd64)\r\n
    Accept: */*\r\n
    Accept-Encoding: gzip\r\n
    Connection: close\r\n
    \r\n
    [Full request URI: http://test.mypythonserver.gg:5002/.well-known/acme-challenge/ZxZaPZD5LtpTD6TBnTysD4jOWzsQvws-sCoWUxNkeZ0]
    [HTTP request 1/1]
    [Response in frame: 120]

After that, I've stopped the Docker images and started again, but set the default IPv4 address for DNS challenges, as said in Pebble's README:

curl -d '{"ip":"127.0.0.1"}' http://localhost:8055/set-default-ipv4

The error has changed:

Get http://test.mypythonserver.gg:5002/.well-known/acme-challenge/P0VJP2GvXyvzLZTw0_qy5FJqVh6N_VadqDLIR9NQr1U: dial tcp 127.0.0.1:5002: connect: connection refused","status":400

Furthermore, there were no HTTP requests captured in Wireshark.

I'm not familiar with Docker, but I thought, that the Docker attempts to call his local IP address, not the address of my original server. So, I've attempted to setup it in bridge network, which subnet was 127.17.0.1/16 (received via docker network inspect bridge), the resulting server setup was python -m http.server 5002 --bind 127.17.0.1 and before that I made curl -d '{"ip":"127.17.0.1"}' http://localhost:8055/set-default-ipv4. The error was the same as in previous case, no HTTP requests captured too.

In each of the following attempts, the server had not received any requests and, obviously, the challenge status was set to invalid.

The other interesting thing I found is that curl 10.30.50.3:5002 successfully connects and returns empty string. Perhaps, I need to somehow add the route for HTTP-01 challenge file in this address?

To sum up, I'm not sure, where do i need to locate my test server or how should i modify the default configuration. I guess, this is not a Java client's issue, because the account is created and the challenge is triggered successfully, but the desired server cannot be located.

OS: Arch Linux x86_64
Kernel: 5.15.13-arch1-1
Wireshark (QT) version: 3.6.1

2 Likes
2

ACME (Pebble, Boulder/LetsEncrypt) needs to connect on port 80 to validate the server. You need to make sure your system is configured to proxy the /.well-known/acme-challenge traffic -- or all traffic -- sent to port 80 onto the python server running on 5002.

That's the first step, now you need to make sure curl localhost/.well-known/acme-challenge/<REQUIRED_NAME> will work. So you'll either have to configure docker to route traffic as needed, or potentially just run a reverse proxy/gateway on port 80.

For debugging this stuff, I use a simple Python server too. The client we released uses it for a configuration tool:
https://github.com/aptise/peter_sslers/blob/main/tools/fake_server.py

and here's a proxy setup for nginx:

I'm not sure what could be triggering the 403 HTTP exception in your stack. That could be on something running on port80.

3 Likes
3

Please show:
cat /etc/hosts

Why?
Because:
test.mypythonserver.gg:5002 with both 10.30.50.3:5002 & 127.0.0.1:5002
Seem inconsistent.
[from some perspectives 10.30.50.3 & 127.0.0.1 may NOT be equally accessible]

3 Likes
4

I've enabled nginx reverse proxy server with the following configuration:

events{}

http {
  server {
    listen        80;
    server_name test.reverseproxy.in;
    
    location  /.well-known/public/whoami  {
      proxy_set_header  X-Real-IP  $remote_addr;
      proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
      proxy_set_header  Host  $host;
      proxy_pass  http://127.0.0.1:5002;
    }	


    location  /.well-known/acme-challenge  {
      proxy_set_header  X-Real-IP  $remote_addr;
      proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
      proxy_set_header  Host  $host;
      proxy_pass  http://127.0.0.1:5002;
    }
  }
}

and ensured that curl localhost/.well-known/acme-challenge/<REQUIRED_NAME> returns correct value, but still getting the same 403 error, as described in first case (also tried with another port, 8333). I investigated behavior of 127.0.0.1:5002 and 10.30.50.3:5002 and ensured that they are completely different. The curl 10.30.50.3:5002/.well-known/acme-challenge/<REQUIRED_NAME> returns empty string and, furthermore, it accepts any route. I tried to use curl 10.30.50.3:5002/anyaddress/1111 and it returns empty string too with 200 OK (captured in Wireshark):

Frame 79: 160 bytes on wire (1280 bits), 160 bytes captured (1280 bits) on interface veth88c85f6, id 0
Ethernet II, Src: 02:42:7a:11:cc:b9 (02:42:7a:11:cc:b9), Dst: 02:42:0a:1e:32:03 (02:42:0a:1e:32:03)
Internet Protocol Version 4, Src: 10.30.50.1, Dst: 10.30.50.3
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 146
    Identification: 0x6f22 (28450)
    Flags: 0x40, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: TCP (6)
    Header Checksum: 0x5304 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.30.50.1
    Destination Address: 10.30.50.3
Transmission Control Protocol, Src Port: 48744, Dst Port: 5002, Seq: 1, Ack: 1, Len: 94
    Source Port: 48744
    Destination Port: 5002
    [Stream index: 3]
    [Conversation completeness: Complete, WITH_DATA (31)]
    [TCP Segment Len: 94]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 2529221429
    [Next Sequence Number: 95    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 781436275
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 502
    [Calculated window size: 64256]
    [Window size scaling factor: 128]
    Checksum: 0x78c4 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (94 bytes)
Hypertext Transfer Protocol
    GET /anyaddress/1111 HTTP/1.1\r\n
    Host: 10.30.50.3:5002\r\n
    User-Agent: curl/7.81.0\r\n
    Accept: */*\r\n
    \r\n
    [Full request URI: http://10.30.50.3:5002/anyaddress/1111]
    [HTTP request 1/1]
    [Response in frame: 81]

Also, checked /etc/hosts, it is empty.
I'm not sure, but according to README the 10.30.50.3:5002 is a challenge server, and, maybe, it is deployed separately (so, the 10.30.50.3:5002 and 127.0.0.1:5002 are different servers). But can't understand, why the destination IP for challenge validation is 10.30.50.3. Probably, i need to change default docker-compose.yml?

2 Likes
5

Yes. Docker needs to expose/route the ports between the container(s).

3 Likes
6

So, just completed the challenge! After several searching, found that "host" network mode for Docker uses host addresses, so the changed config is:

version: '3'
services:
  pebble:
    image: letsencrypt/pebble:latest
    command: pebble -config /test/config/pebble-config.json -strict -dnsserver 127.0.0.1:8053
    network_mode: "host"
    ports:
      - 14000:14000  # HTTPS ACME API
      - 15000:15000  # HTTPS Management API
  challtestsrv:
    image: letsencrypt/pebble-challtestsrv:latest
    command: pebble-challtestsrv -defaultIPv6 "" -defaultIPv4 127.0.0.1 -http01 ""
    network_mode: "host"
    ports:
      - 8055:8055  # HTTP Management API

Here I also added -http01 "" to disable challenge server creation (because i create it myself). The test python server should be on port 5002 (python -m http.server 5002 --bind 127.0.0.1).

After that, I've successfully received several GET requests and completed the challenge. I guess, it can be done better with reverse proxies and Docker routing settings, but as a simple testing environment it works well. Thanks for the help!

3 Likes
7

Can't you just leave challtestsrv out entirely if you don't need it?

3 Likes
8

If you understand how/why this setup works, then that's the best option for your environment!

2 Likes
9

You show:

And then

I'm confused :confused:

3 Likes
10

Good point, disabled it. Thanks!

3 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.