--dry-run succeeds but without it, it fails

Help
1

I am trying to use a CNAME record instead of an A record, so I have the following setup:

  • On domain mktg.dev I have created a CNAME record k2mac.mktg.dev pointing to mktg-dev.k2mac.nl
  • On mktg-dev.k2mac.nl I have an A record pointing to the server IP: 3.66.118.132
  • On the host with IP: 3.66.118.132 there is an HAProxy using the http-01 method to obtain and renew certs.

This host runs other domains and has zero issues with the issuance of certificates when using A records, but for CNAME records it only works when the --dry-run parameter is enabled and then it fails without the --dry-run parameter:

root@hap-4-10 ~ # certbot certonly --standalone -d k2mac.mktg.dev --non-interactive --agree-tos --email yosu.cadilla@gmail.com --http-01-port=8888 --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate

IMPORTANT NOTES:

  • The dry run was successful.

root@hap-4-10 ~ # certbot certonly --standalone -d k2mac.mktg.dev --non-interactive --agree-tos --email yosu.cadilla@gmail.com --http-01-port=8888
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for k2mac.mktg.dev
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. k2mac.mktg.dev (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: 3.66.118.132: Fetching http://k2mac.mktg.dev/.well-known/acme-challenge/EPTBf8klz9ahNQaBmCugzxUDo4lCL02g8KNMuqluNM0: Connection reset by peer

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: k2mac.mktg.dev
    Type: connection
    Detail: 3.66.118.132: Fetching
    http://k2mac.mktg.dev/.well-known/acme-challenge/EPTBf8klz9ahNQaBmCugzxUDo4lCL02g8KNMuqluNM0:
    Connection reset by peer

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you're using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

Very confused by this result, any pointers greatly appreciated.

2

@MikeMcQ Could you perhaps do your Paulo Alto (or what's it called) magic? :stuck_out_tongue:

4 Likes
4

Oh, never mind my prior post. I did not read the op close enough.

@Yosu How do you know to route http requests from HAProxy to the certbot running in standalone on port 8888?

I get a "connection reset by peer" for any request to that domain. Should I right now?

curl -i4 http://k2mac.mktg.dev
curl: (56) Recv failure: Connection reset by peer

curl -i http://k2mac.mktg.dev/SomeTestPage
curl: (56) Recv failure: Connection reset by peer

Debugging standalone is harder. It sometimes helps to add --debug-challenges -v to the certbot command. This will pause certbot and wait for a challenge. We can then test comms to it if you leave it paused.

We can try that but let us know about your routing for http requests to your standalone on port 8888

3 Likes
5

Hi @MikeMcQ,

Thanks for loking into this issue.

This is how HAP is configured:

frontend http
    bind *:80 accept-proxy
    bind *:443 ssl crt /etc/ssl/hapcerts/ accept-proxy alpn h2,http/1.1
    mode http
    capture request header X-Forwarded-For len 15
    http-request set-header X-Forwarded-Proto https


# ACL Certbot
    acl certbot path_beg /.well-known/acme-challenge/
    use_backend certbot if certbot

#####   DEFAULT BACKENDS   #####

    backend certbot
    server certbot 127.0.0.1:8888

I have removed all the parameters unrelated to certbot...

This server hosts several domains, each one has a certificate previously issued from this HAProxy server, everything runs smoothly, always, for years, if we use an A record.

It is now that I am testing CNAME records that the issue appears. When issuing the command with --dry-run, it works perfectly.
What is the difference when running a command with or without this parameter?

1 Like
6

I'm not that familiar with HAProxy, but is accept-proxy enough to make Certbot work? As it'll be using HTTP..

Although that doesn't explain why a dry run would suddenly succeed..

3 Likes
7

The --dry-run success is probably a cached authorization + really old version of Certbot.

I agree that the accept-proxy is probably the culprit. I think that changes the protocol from HTTP to the Proxy protocol. Just bind *:80 should be fine.

5 Likes
8

Debugging standalone is harder. It sometimes helps to add --debug-challenges -v to the certbot command.

Thanks for the tip, running the command with those parameters I have found this:

Performing the following challenges:
http-01 challenge for k2mac.mktg.dev
Successfully bound to :8888 using IPv6
Certbot wasn't able to bind to :8888 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
Waiting for verification...

Do you think it could be the problem?

1 Like
9

I don't think the binding warning is a problem. I still get 'reset by peer" for any request. I should see something else if http request was reaching the certbot standalone while it is waiting

curl -i http://k2mac.mktg.dev/MikeTest
curl: (56) Recv failure: Connection reset by peer

curl -i http://k2mac.mktg.dev/.well-known/acme-challenge/MikeTest2
curl: (56) Recv failure: Connection reset by peer

You should try what _az suggests

4 Likes
10

That was probably the issue, thanks a lot @_az

root@hap-4-10 ~ # apt list --upgradable
Listing... Done
certbot/bionic 0.31.0-2~deb10u1+ubuntu18.04.1+certbot+3 all [upgradable from: 0.31.0-1+ubuntu18.04.1+certbot+1]
haproxy/bionic 2.1.12-1ppa1~bionic amd64 [upgradable from: 2.1.5-1ppa1~bionic]
libsqlite3-0/bionic-updates,bionic-security 3.22.0-1ubuntu0.7 amd64 [upgradable from: 3.22.0-1ubuntu0.6]
python3-certbot/bionic 0.31.0-2~deb10u1+ubuntu18.04.1+certbot+3 all [upgradable from: 0.31.0-1+ubuntu18.04.1+certbot+1]
python3-configobj/bionic 5.0.6-2+ubuntu18.04.1+certbot+1 all [upgradable from: 5.0.6-2]
python3-zope.interface/bionic 4.3.2-1+ubuntu18.04.1+certbot+1 amd64 [upgradable from: 4.3.2-1build2]
ubuntu-advantage-tools/bionic-updates 27.11.3~18.04.1 amd64 [upgradable from: 27.11.2~18.04.1]

Upgrading the packages seems to have solved the problem.

@MikeMcQ
About the "reset by peer" messages: There was no actual server/service listening on the destination specified on HAProxy, so I added an apache server with an app.

Now I am not sure what fixed the problem was the package upgrade or the valid destination.
Anyhow, I have got a valid certificate, thank you very much to the both of you!!

2 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.