Need Help, Please

Help
1

Hi Everybody (Hi, Dr. Nik - Simpsons’ reference :stuck_out_tongue: )

People, please read all of the notes, logs, and and other information - I’ve notices that some repliers, while trying to be helpful, which is appreciated, have a tendance to jump in and suggest things that have either been covered already or are not possible (for a varienty of reasons) and so are not viable/valid solutions/suggestions. grin:

My domainins are: peregrineit.net, matthewj.black

I ran these commands:

  1. certbot certonly --dry-run --agree-tos --standalone --preferred-challenges http --http-01-port 54321 -d www.peregrineit.net
  2. certbot certonly --dry-run --agree-tos --standalone --preferred-challenges http --http-01-port 54321 -d www.matthewj.black

They produced this output:

  1. Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator standalone, Installer None
    Enter email address (used for urgent renewal and security notices) (Enter ‘c’ to
    cancel): matthew@peregrineit.net
    Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
    Obtaining a new certificate
    Performing the following challenges:
    http-01 challenge for www.peregrineit.net
    Waiting for verification…
    Challenge failed for domain www.peregrineit.net
    http-01 challenge for www.peregrineit.net
    Cleaning up challenges
    Some challenges have failed.

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: www.peregrineit.net
    Type: connection
    Detail: Fetching
    http://www.peregrineit.net/.well-known/acme-challenge/B6V1lgTaFwTkOgkwnZODnV4HdbDjNgNDyLwvbFpvN5s:
    Error getting validation data

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

  • Your account credentials have been saved in your Certbot
    configuration directory at /etc/letsencrypt. You should make a
    secure backup of this folder now. This configuration directory will
    also contain certificates and private keys obtained by Certbot so
    making regular backups of this folder is ideal.

  1. Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator standalone, Installer None
    Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
    Obtaining a new certificate
    Performing the following challenges:
    http-01 challenge for www.matthewj.black
    Waiting for verification…
    Challenge failed for domain www.matthewj.black
    http-01 challenge for www.matthewj.black
    Cleaning up challenges
    Some challenges have failed.

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: www.matthewj.black
    Type: connection
    Detail: Fetching
    http://www.matthewj.black/.well-known/acme-challenge/g9jT6Asif9y7kRK0B93i9vWRbq53cFPSJ7lS6HnifeM:
    Error getting validation data

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

My web server stack is (include version): Gateway (Router) => HA-Proxy version 1.8.22 2019/10/25 => Nginx version: nginx/1.17.6

The operating system all my servers run on is (include version): CentOS Linux release 7.7.1908 (Core)

My hosting provider, if applicable, is: Self Hosting

I can login to a root shell on my machine (yes or no, or I don’t know): Yes, All of them

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.39.0

Other Important Informtion:

  • Both the websites are running on the same Nginx box.
  • Everything works re: accessing both webservers - try it, you’ll see (via http, not https).
  • Certbot is running on the HAProxy Box on port :54321.
  • The Gateway redirects to the HAProxy Box, and only allows port :80 (at the moment).
  • Everything flows through the HAProxy Box.
  • The HAProxy Box is only listening on port :80.
  • The HAProxy Box redirects to the Certbot Backend if it hears “/.well-known/acme-challenge/”, otherwise it uses the default backend of the Nginx Server.
  • The stack cannot be changed.
  • Everything was working, including certificates, before an “incident” occurred which meant that the HAProxy Box had to be rebuilt (backups were also corrupted/losts).

The haproxy.cfg file is:
++++++++++++++++++++++++++++++
1 frontend hap
2 acl lerequest path_beg -i /.well-known/acme-challenge/
3 bind *:80
4 use_backend certbot if lerequest
5 default_backend nginx
6
7 backend nginx
8 server nginx 10.99.99.98:80
9
10 backend certbot
11server certbot 127.0.0.1:54321
++++++++++++++++++++++++++++++

The Debug Log file for the www.matthewj.black run is (the www.peregrineit.net run was the same/similar):
++++++++++++++++++++++++++++++
1 2019-11-28 21:44:13,000:DEBUG:certbot.main:certbot version: 0.39.0
2 2019-11-28 21:44:13,000:DEBUG:certbot.main:Arguments: [’–dry-run’, ‘–agree-tos’, ‘–standalone’, ‘–preferred-challenges’, ‘http’, ‘–http-01-port’, ‘54321’, ‘-d’, ‘www.matthewj.black’]
3 2019-11-28 21:44:13,000:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null, PluginEntryPoint#standalone,PluginEntryPoint#webroot)
4 2019-11-28 21:44:13,053:DEBUG:certbot.log:Root logging level set at 20
5 2019-11-28 21:44:13,053:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
6 2019-11-28 21:44:13,055:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
7 2019-11-28 21:44:13,057:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
8 Description: Spin up a temporary webserver
9 Interfaces: IAuthenticator, IPlugin
10 Entry point: standalone = certbot.plugins.standalone:Authenticator
11 Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f3fc1d86490>
12 Prep: True
13 2019-11-28 21:44:13,057:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7f3fc1d86490> and installer None
14 2019-11-28 21:44:13,057:INFO:certbot.plugins.selection:Plugins selected: Authenticator standalone, Installer None
15 2019-11-28 21:44:13,106:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u’https://acme-staging-v02.api.letsencrypt.org/acme/acct/10037040’, new_authzr_uri=None, terms_of_service=None), edb9d3f61c434fe43ee12fbb30ed00c5, Meta(creation_host=u’hap01.mjb-co.pri’, creation_dt=datetime.datetime(2019, 7, 14, 5, 55, 20, tzinfo=)))>
16 2019-11-28 21:44:13,149:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
17 2019-11-28 21:44:13,163:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
18 2019-11-28 21:44:14,347:DEBUG:requests.packages.urllib3.connectionpool:“GET /directory HTTP/1.1” 200 724
19 2019-11-28 21:44:14,348:DEBUG:acme.client:Received response:
20 HTTP 200
21 content-length: 724
22 strict-transport-security: max-age=604800
23 server: nginx
24 connection: keep-alive
25 cache-control: public, max-age=0, no-cache
26 date: Thu, 28 Nov 2019 10:44:14 GMT
27 x-frame-options: DENY
28 content-type: application/json
29
30 {
31 “keyChange”: “https://acme-staging-v02.api.letsencrypt.org/acme/key-change”,
32 “meta”: {
33 “caaIdentities”: [
34 “letsencrypt.org”
35 ],
36 “termsOfService”: “https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf”,
37 “website”: “https://letsencrypt.org/docs/staging-environment/”
38 },
39 “newAccount”: “https://acme-staging-v02.api.letsencrypt.org/acme/new-acct”,
40 “newNonce”: “https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce”,
41 “newOrder”: “https://acme-staging-v02.api.letsencrypt.org/acme/new-order”,
42 “revokeCert”: “https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert”,
43 “ry1lbkjnL60”: “Adding random entries to the directory”
44 }
45 2019-11-28 21:44:14,351:INFO:certbot.main:Obtaining a new certificate
46 2019-11-28 21:44:14,419:DEBUG:acme.client:Requesting fresh nonce
47 2019-11-28 21:44:14,419:DEBUG:acme.client:Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
48 2019-11-28 21:44:14,711:DEBUG:requests.packages.urllib3.connectionpool:“HEAD /acme/new-nonce HTTP/1.1” 200 0
49 2019-11-28 21:44:14,712:DEBUG:acme.client:Received response:
50 HTTP 200
51 strict-transport-security: max-age=604800
52 server: nginx
53 connection: keep-alive
54 link: https://acme-staging-v02.api.letsencrypt.org/directory;rel=“index”
55 cache-control: public, max-age=0, no-cache
56 date: Thu, 28 Nov 2019 10:44:14 GMT
57 x-frame-options: DENY
58 replay-nonce: 0001xX19NI6vNnBawgGWHDGm25–zFSIsBbGqirvTM6y2W4
59
60
61 2019-11-28 21:44:14,713:DEBUG:acme.client:Storing nonce: 0001xX19NI6vNnBawgGWHDGm25–zFSIsBbGqirvTM6y2W4
62 2019-11-28 21:44:14,714:DEBUG:acme.client:JWS payload:
63 {
64 “identifiers”: [
65 {
66 “type”: “dns”,
67 “value”: “www.matthewj.black”
68 }
69 ]
70 }
71 2019-11-28 21:44:14,719:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
72 {
73 “protected”: “eyJub25jZSI6ICIwMDAxeFgxOU5JNnZObkJhd2dHV0hER20yNS0tekZTSXNCYkdxaXJ2VE02eTJXNCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xMDAzNzA0MCIsICJhbGciOiAiUlMyNTYifQ”,
74 “payload”: “ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwgCiAgICAgICJ2YWx1ZSI6ICJ3d3cubWF0dGhld2ouYmxhY2siCiAgICB9CiAgXQp9”,
75 “signature”: “Aqv_GFGo303I9ZW2b3pyn-YbvyuQ9XoaHvjQlATIxP_OB6pDeAV8USCwqxbduN_oNJJvUC7aNaTAqVMHSpK6BvCoein7dT7tVNmKzoRvzvuTEEf9yrQ3o4CEHkuJZDwXUb74As3Wsh1YAvcCqwgvSK473FDDhBP5uM82zp6eG6W-xrX-MH4KUYW-jAq9IGcSgo6ay9Z3eVvUavHIigLj0264GbWTRWLhTi-dLs087WD4_fczJIOn9cINdo03yYgdtC6xJ2Nj6-IXfW0i74YDXRpoovhUOjwGlE14cu_PS8wroi0J75URIUM1k7JYGNQsSRF641E5S7AJVKYh3rH06Q”
76 }
77 2019-11-28 21:44:15,039:DEBUG:requests.packages.urllib3.connectionpool:“POST /acme/new-order HTTP/1.1” 201 360
78 2019-11-28 21:44:15,040:DEBUG:acme.client:Received response:
79 HTTP 201
80 content-length: 360
81 cache-control: public, max-age=0, no-cache
82 strict-transport-security: max-age=604800
83 server: nginx
84 connection: keep-alive
85 link: https://acme-staging-v02.api.letsencrypt.org/directory;rel=“index”
86 location: https://acme-staging-v02.api.letsencrypt.org/acme/order/10037040/63502785
87 boulder-requester: 10037040
88 date: Thu, 28 Nov 2019 10:44:14 GMT
89 x-frame-options: DENY
90 content-type: application/json
91 replay-nonce: 0001FlN2EdDDwK5EP0hKNMdRojoB6fFmHWT0aXyHCUqvOgI
92
93 {
94 “status”: “pending”,
95 “expires”: “2019-12-05T10:44:14.934726806Z”,
96 “identifiers”: [
97 {
98 “type”: “dns”,
99 “value”: “www.matthewj.black”
100 }
101 ],
102 “authorizations”: [
103 “https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/23857232”
104 ],
105 “finalize”: “https://acme-staging-v02.api.letsencrypt.org/acme/finalize/10037040/63502785”
106 }
107 2019-11-28 21:44:15,041:DEBUG:acme.client:Storing nonce: 0001FlN2EdDDwK5EP0hKNMdRojoB6fFmHWT0aXyHCUqvOgI
108 2019-11-28 21:44:15,041:DEBUG:acme.client:JWS payload:
109
110 2019-11-28 21:44:15,045:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/23857232:
111 {
112 “protected”: “eyJub25jZSI6ICIwMDAxRmxOMkVkRER3SzVFUDBoS05NZFJvam9CNmZGbUhXVDBhWHlIQ1Vxdk9nSSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8yMzg1NzIzMiIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzEwMDM3MDQwIiwgImFsZyI6ICJSUzI1NiJ9”,
113 “payload”: “”,
114 “signature”: “o0e1IX4vZr7AHR32MpXzyDN8SsLXhD936ItLd5xU6RGip6nzGNUZayozTDPI05PJu8AvKoJvewrh_50NfZI2im1PgPFnQYHaYwYf_WhL0qWLWoc1Tisvnz1aqjAii_tPA9Y4OVEEbcRHSeWqOfGjP4K9lvJZ1IKrGXfAgjXUtIz7CcDgIgdSOd2ySfYoSGyJqjUHZoj8l6b0j740h46g1Q9ZfkLVwFJvCEHNBgHWwT2h7_wb0-Qy1E6t66JmYx5xt1JdBFL9c3k9ttWc4CIV624d7SwLKOvZfInXRvWqaYq35Hghj_3YMTincAzOw09hc0nR4Vs0SulkwpRadHXi5w”
115 }
116 2019-11-28 21:44:15,341:DEBUG:requests.packages.urllib3.connectionpool:“POST /acme/authz-v3/23857232 HTTP/1.1” 200 814
117 2019-11-28 21:44:15,342:DEBUG:acme.client:Received response:
118 HTTP 200
119 content-length: 814
120 cache-control: public, max-age=0, no-cache
121 strict-transport-security: max-age=604800
122 server: nginx
123 connection: keep-alive
124 link: https://acme-staging-v02.api.letsencrypt.org/directory;rel=“index”
125 boulder-requester: 10037040
126 date: Thu, 28 Nov 2019 10:44:15 GMT
127 x-frame-options: DENY
128 content-type: application/json
129 replay-nonce: 0002hZsejl00JJW_INxQClHlZtoNrUJnFznSPrQIdGwYjCc
130
131 {
132 “identifier”: {
133 “type”: “dns”,
134 “value”: “www.matthewj.black”
135 },
136 “status”: “pending”,
137 “expires”: “2019-12-05T10:44:14Z”,
138 “challenges”: [
139 {
140 “type”: “http-01”,
141 “status”: “pending”,
142 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/AfUr0A”,
143 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”
144 },
145 {
146 “type”: “dns-01”,
147 “status”: “pending”,
148 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/WaPkvQ”,
149 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”
150 },
151 {
152 “type”: “tls-alpn-01”,
153 “status”: “pending”,
154 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/W4QSsQ”,
155 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”
156 }
157 ]
158 }
159 2019-11-28 21:44:15,342:DEBUG:acme.client:Storing nonce: 0002hZsejl00JJW_INxQClHlZtoNrUJnFznSPrQIdGwYjCc
160 2019-11-28 21:44:15,344:INFO:certbot.auth_handler:Performing the following challenges:
161 2019-11-28 21:44:15,344:INFO:certbot.auth_handler:http-01 challenge for www.matthewj.black
162 2019-11-28 21:44:15,353:DEBUG:acme.standalone:Failed to bind to :54321 using IPv6
163 2019-11-28 21:44:15,359:DEBUG:acme.standalone:Successfully bound to :54321 using IPv4
164 2019-11-28 21:44:15,367:INFO:certbot.auth_handler:Waiting for verification…
165 2019-11-28 21:44:15,369:DEBUG:acme.client:JWS payload:
166 {
167 “type”: “http-01”,
168 “resource”: “challenge”
169 }
170 2019-11-28 21:44:15,373:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/AfUr0A:
171 {
172 “protected”: “eyJub25jZSI6ICIwMDAyaFpzZWpsMDBKSldfSU54UUNsSGxadG9OclVKbkZ6blNQclFJZEd3WWpDYyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9jaGFsbC12My8yMzg1NzIzMi9BZlVyMEEiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xMDAzNzA0MCIsICJhbGciOiAiUlMyNTYifQ”,
173 “payload”: “ewogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9”,
174 “signature”: “X_DE1LQk9eykC8dg3gvvw1C_atJyiBbCO3K7Jw1e5_hwu18_JpO3Bbn8JYP-T4N2jKjQMTYVlameZrZ73bFAtRyDtsvuwB2mw2lZPaKdzhBlApdy2xysgKTzX93atxbrsoFgjq-bHRLToJF0HGvqJSGnb4YhsCKlXntonKIy-gW8HB5q6fiJiuLF_iMuCth0oh9tajLn_A50D87MpBYolleVxtHaB64XRAEKNTwCMKTebqwKVeW_ScJExf63-qv8yWcnT0WcLFSVuP9YR0JRq_YuGGprR2Ca7R__E2-WZubTkVfJtKQTHdOjOnhFgja4yfX9x8yoNYU6jh_-o86cmg”
175 }
176 2019-11-28 21:44:15,670:DEBUG:requests.packages.urllib3.connectionpool:“POST /acme/chall-v3/23857232/AfUr0A HTTP/1.1” 200 19 1
177 2019-11-28 21:44:15,671:DEBUG:acme.client:Received response:
178 HTTP 200
179 content-length: 191
180 cache-control: public, max-age=0, no-cache
181 strict-transport-security: max-age=604800
182 server: nginx
183 connection: keep-alive
184 link: https://acme-staging-v02.api.letsencrypt.org/directory;rel=“index”, <https://acme-staging-v02.api.letsencrypt.org/ac me/authz-v3/23857232>;rel=“up”
185 location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/AfUr0A
186 boulder-requester: 10037040
187 date: Thu, 28 Nov 2019 10:44:15 GMT
188 x-frame-options: DENY
189 content-type: application/json
190 replay-nonce: 0002KxPsRVZReTXUkrCvttz-uI_5zT7jhQPS15gBA0GgSRs
191
192 {
193 “type”: “http-01”,
194 “status”: “pending”,
195 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/AfUr0A”,
196 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”
197 }
198 2019-11-28 21:44:15,672:DEBUG:acme.client:Storing nonce: 0002KxPsRVZReTXUkrCvttz-uI_5zT7jhQPS15gBA0GgSRs
199 2019-11-28 21:44:16,673:DEBUG:acme.client:JWS payload:
200
201 2019-11-28 21:44:16,679:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3 /23857232:
202 {
203 “protected”: “eyJub25jZSI6ICIwMDAyS3hQc1JWWlJlVFhVa3JDdnR0ei11SV81elQ3amhRUFMxNWdCQTBHZ1NScyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8yMzg1NzIzMiIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzEwMDM3MDQwIiwgImFsZyI6ICJSUzI1NiJ9”,
204 “payload”: “”,
205 “signature”: “yxkbVoOV009SBz5hQIKQlXrrwtyeGbU03yF_VoG0et-UoOEeBZeH3QlSA-moirHexrFbyxSLA3-JrpNDJJKcxCv7HINOFyzRx-lBB_B6r-7YCvSz6McesyxhGlO8vR0NdVfD5-BTbjFNESA-ldDc2UbFpxgxwZuOthbBf5qtOkfPPzQtMntAuusUQ1Ee1MNX6uut6RfrUqagHfFNsoCn8U5nOPyDr4A2lsKm3uVLt7oRKrlCRhuuhNiE0GiZe1tM9bT0SF_9SiwdtuZBgB1OIhfhw3iZln8Vm4Cz2gQmGQg1-nxYiOjvgPx2i5grfWXw6u-Gls4cWXPNsPLvwg1HIQ”
206 }
207 2019-11-28 21:44:16,978:DEBUG:requests.packages.urllib3.connectionpool:“POST /acme/authz-v3/23857232 HTTP/1.1” 200 814
208 2019-11-28 21:44:16,978:DEBUG:acme.client:Received response:
209 HTTP 200
210 content-length: 814
211 cache-control: public, max-age=0, no-cache
212 strict-transport-security: max-age=604800
213 server: nginx
214 connection: keep-alive
215 link: https://acme-staging-v02.api.letsencrypt.org/directory;rel=“index”
216 boulder-requester: 10037040
217 date: Thu, 28 Nov 2019 10:44:16 GMT
218 x-frame-options: DENY
219 content-type: application/json
220 replay-nonce: 0001A5idXFD82Z6JJJ1IG370jlJDoWC3Gq8ifZ4A4QULlFs
221
222 {
223 “identifier”: {
224 “type”: “dns”,
225 “value”: “www.matthewj.black”
226 },
227 “status”: “pending”,
228 “expires”: “2019-12-05T10:44:14Z”,
229 “challenges”: [
230 {
231 “type”: “http-01”,
232 “status”: “pending”,
233 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/AfUr0A”,
234 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”
235 },
236 {
237 “type”: “dns-01”,
238 “status”: “pending”,
239 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/WaPkvQ”,
240 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”
241 },
242 {
243 “type”: “tls-alpn-01”,
244 “status”: “pending”,
245 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/W4QSsQ”,
246 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”
247 }
248 ]
249 }
250 2019-11-28 21:44:16,979:DEBUG:acme.client:Storing nonce: 0001A5idXFD82Z6JJJ1IG370jlJDoWC3Gq8ifZ4A4QULlFs
251 2019-11-28 21:44:19,981:DEBUG:acme.client:JWS payload:
252
253 2019-11-28 21:44:19,987:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/23857232:
254 {
255 “protected”: “eyJub25jZSI6ICIwMDAxQTVpZFhGRDgyWjZKSkoxSUczNzBqbEpEb1dDM0dxOGlmWjRBNFFVTGxGcyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8yMzg1NzIzMiIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzEwMDM3MDQwIiwgImFsZyI6ICJSUzI1NiJ9”,
256 “payload”: “”,
257 “signature”: “uSlQMt7_fW6xzF4UQOAQlYRLpVESVpxGA12KkqVGkK02a6pOKptMbrQdNqx_PnVgn4UR0VlU6P9DW4IfTIqmT2F0XsPvLXhFemtlEnbBgcIJCZIUMtL87D0oKixz8WJFVIMkCG93a6O7dPeXpt-85fNWrT4LjocS8aDeYZeBpzmrPcDcjAxRWEQtoeAytC6Ghet7G_AuNXdonORoyb79mHqULE_jo-rdfH9dGvHp1-nbEqY6xgsq14YUFxTz9ToXTTi9SPp9JWwvAwiiUYLpZ_JJCwWgo3nxqi7drJCoCSQoCy6GHQ5Zt8bD_81sIGWMQ8169JkSgm1krIu_tgQM-Q”
258 }
259 2019-11-28 21:44:20,284:DEBUG:requests.packages.urllib3.connectionpool:“POST /acme/authz-v3/23857232 HTTP/1.1” 200 1281
260 2019-11-28 21:44:20,285:DEBUG:acme.client:Received response:
261 HTTP 200
262 content-length: 1281
263 cache-control: public, max-age=0, no-cache
264 strict-transport-security: max-age=604800
265 server: nginx
266 connection: keep-alive
267 link: https://acme-staging-v02.api.letsencrypt.org/directory;rel=“index”
268 boulder-requester: 10037040
269 date: Thu, 28 Nov 2019 10:44:20 GMT
270 x-frame-options: DENY
271 content-type: application/json
272 replay-nonce: 0001xJNNGTN4jM9P5LCnQUYMh-nQOQFMdZKYYgo8QBA-jQs
273
274 {
275 “identifier”: {
276 “type”: “dns”,
277 “value”: “www.matthewj.black”
278 },
279 “status”: “invalid”,
280 “expires”: “2019-12-05T10:44:14Z”,
281 “challenges”: [
282 {
283 “type”: “http-01”,
284 “status”: “invalid”,
285 “error”: {
286 “type”: “urn:ietf:params:acme:error:unauthorized”,
287 “detail”: “Invalid response from http://www.matthewj.black/.well-known/acme-challenge/Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA [218.214.86.96]: “\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e403 Forbidden\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody\u003e\r\n\u003ccenter\u003e\u003ch1\u003e403 Forbidden\u003c/h1\u003e\u003c/center\u003e\r\ n\u003chr\u003e\u003ccenter\u003enginx\u003c/center\u003e\r\n””,
288 “status”: 403
289 },
290 “url”: “https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/AfUr0A”,
291 “token”: “Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”,
292 “validationRecord”: [
293 {
294 “url”: “http://www.matthewj.black/.well-known/acme-challenge/Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA”,
295 “hostname”: “www.matthewj.black”,
296 “port”: “80”,
297 “addressesResolved”: [
298 “218.214.86.96”
299 ],
300 “addressUsed”: “218.214.86.96”
301 }
302 ]
303 }
304 ]
305 }
306 2019-11-28 21:44:20,285:DEBUG:acme.client:Storing nonce: 0001xJNNGTN4jM9P5LCnQUYMh-nQOQFMdZKYYgo8QBA-jQs
307 2019-11-28 21:44:20,286:WARNING:certbot.auth_handler:Challenge failed for domain www.matthewj.black
308 2019-11-28 21:44:20,286:INFO:certbot.auth_handler:http-01 challenge for www.matthewj.black
309 2019-11-28 21:44:20,287:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:
310
311 Domain: www.matthewj.black
312 Type: unauthorized
313 Detail: Invalid response from http://www.matthewj.black/.well-known/acme-challenge/Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA [218.214.86.96]: “\r\n403 Forbidden\r\n\r\n

403 Forbidden

\r\n
nginx\r\n”
314
315 To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that doma in contain(s) the right IP address.
316 2019-11-28 21:44:20,288:DEBUG:certbot.error_handler:Encountered exception:
317 Traceback (most recent call last):
318 File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 90, in handle_authorizations
319 self._poll_authorizations(authzrs, max_retries, best_effort)
320 File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 154, in _poll_authorizations
321 raise errors.AuthorizationError(‘Some challenges have failed.’)
322 AuthorizationError: Some challenges have failed.
323
324 2019-11-28 21:44:20,288:DEBUG:certbot.error_handler:Calling registered functions
325 2019-11-28 21:44:20,288:INFO:certbot.auth_handler:Cleaning up challenges
326 2019-11-28 21:44:20,289:DEBUG:certbot.plugins.standalone:Stopping server at 0.0.0.0:54321…
327 2019-11-28 21:44:20,366:DEBUG:certbot.log:Exiting abnormally:
328 Traceback (most recent call last):
329 File “/bin/certbot”, line 9, in
330 load_entry_point(‘certbot==0.39.0’, ‘console_scripts’, ‘certbot’)()
331 File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 1378, in main
332 return config.func(config, plugins)
333 File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 1265, in certonly
334 lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
335 File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 121, in _get_and_save_cert
336 lineage = le_client.obtain_and_enroll_certificate(domains, certname)
337 File “/usr/lib/python2.7/site-packages/certbot/client.py”, line 405, in obtain_and_enroll_certificate
338 cert, chain, key, _ = self.obtain_certificate(domains)
339 File “/usr/lib/python2.7/site-packages/certbot/client.py”, line 348, in obtain_certificate
340 orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
341 File “/usr/lib/python2.7/site-packages/certbot/client.py”, line 384, in _get_order_and_authorizations
342 authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
343 File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 90, in handle_authorizations
344 self._poll_authorizations(authzrs, max_retries, best_effort)
345 File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 154, in _poll_authorizations
346 raise errors.AuthorizationError(‘Some challenges have failed.’)
347 AuthorizationError: Some challenges have failed.
++++++++++++++++++++++++++++++
2

I'm no HAProxy user, but for some reason, this lines don't seem to work.

If you look at the exact error from the debug messages and the challenge output (https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/23857232/AfUr0A), you can see the Let's Encrypt authentication server gets a reply from nginx, not from certbot:

"detail": "Invalid response from http://www.matthewj.black/.well-known/acme-challenge/Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA [218.214.86.96]: \"\u003chtml\u003e\\r\\n\u003chead\u003e\u003ctitle\u003e403 Forbidden\u003c/title\u003e\u003c/head\u003e\\r\\n\u003cbody\u003e\\r\\n\u003ccenter\u003e\u003ch1\u003e403 Forbidden\u003c/h1\u003e\u003c/center\u003e\\r\\n\u003chr\u003e\u003ccenter\u003enginx\u003c/center\u003e\\r\\n\""

Compare the above error message with the actual challenge URL: view-source:http://www.matthewj.black/.well-known/acme-challenge/Uj675UmROXmGNABY_ez9ZkHqdIQa8OP9udcrd2APNaA

Looks identical to me.

So your requests aren't hitting the certbot server, but nginx. So I assume your HAProxy configuration fails for some reason.

1 Like
3

Hi Osiris, thanks for replying

OK, so unless I’ve made a major Boo-Boo, those two lines from the HAProxy.cfg are correct: according to just about all the websites out there on the topic of haproxy/letsencrypt, and from our hard-copy configuration notes from when we did have the system working (but I could be wrong).

Also, what makes you say the reply is coming from nginx? I can’t see anything in the quoted line to indicate this?

Cheers

4

It says "nginx" in the reply... Near the end.

Try debugging HAProxy with its logs while trying to validate a challenge.

1 Like
5

Oh, so it does - “enginx”

Trouble is, its “nginx” not “enginx” - unless the “e” belongs to the “u003” :grin:

OK, so based on that, does anyone with HAProxy experience want to tell me what’s wrong? :grinning:

1 Like
6

The e is indeed part of the unicode character U+003E.

Like I said:

1 Like
7

Right, so, after going over the haproxy logs it turns out the that HAProxy is directing the CertBot request correctly ie the Certbot Request is going to the Certbot backend.

BUT certbot is refusing the connection and replying with a 503.

So it appears (to me at least) to be a Certbot issue - or maybe a file permissions isssue, maybe - but I can’t think of anything else.

Anyone?

HAProxy Log:

Dec 3 17:34:54 haproxy_box haproxy[7678]: 54.245.186.160:38012 [03/Dec/2019:17:34:54.858] hap_frontend certbot_backend/certbot 0/0/-1/-1/0 503 212 - - SC-- 1/1/0/0/3 0/0 “GET /.well-known/acme-challen ge/zw4uKSLne_fwHeOXKoMR80ZJlLUKC4_nGCV0IiSWeAU HTTP/1.1”

8

certbot doesn't seem to be able to generate HTTP 503 responses according to a quick search in the code.

Perhaps you can debug the connection between HAProxy and certbot more? Perhaps tcpdump the attempt?

1 Like
9

At what time are you checking this? If Certbot is not running the validation server at the time of the request, a 503 is normal and expected.

I have letsdebug.net setup the same way, so if you randomly visit http://letsdebug.net/.well-known/acme-challenge/test , haproxy reports a 503 because the Certbot standalone server does not exist except for the short time when you actually run Certbot.

This is not the error that Let's Encrypt would report, if your haproxy server was reporting a 503. "Error getting validation data" is pretty much a fallback reserved for protocol and some networking issues.

I would be curious to see what error Let's Encrypt reports now, when you try run certbot certonly. It should be something different. Or perhaps it will succeed - I don't see anything particularly wrong in your setup.

Here is one debugging technique that can be useful.

Run this command, interactively:

certbot certonly --dry-run --standalone \
--preferred-challenges http --http-01-port 54321 \
-d www.peregrineit.net -d foobar.example.com \
--debug-challenges

Note, that the presence of foobar.example.com is important. Do not remove it.

Certbot should pause with a message like:

Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.

At this time, you should be able to make requests to your domains under the /.well-known/acme-challenge/ path, and debug the connectivity between haproxy and Certbot.

It shouldn't 503 and the Certbot standalone server won't disappear until you continue the above prompt.

1 Like
10

Will run the debug as you suggest - but will also point out the cert bot IS running: see post #1 for the command that was run. :slight_smile:

11

So, here's the new command and the resulting output:

certbot certonly --dry-run --standalone --preferred-challenges http --http-01-port 54321 -d www.peregrineit.net -d foobar.example.com --debug-challenges

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for foobar.example.com
http-01 challenge for www.peregrineit.net
Waiting for verification...

Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.

Press Enter to Continue

Challenge failed for domain foobar.example.com
Challenge failed for domain www.peregrineit.net
http-01 challenge for foobar.example.com
http-01 challenge for www.peregrineit.net
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

12

What I meant to suggest was to try debug it by hand.

At this point, don't press Enter.

In a new terminal, try investigate what happens when you try to route the request via haproxy, or try request it from Certbot directly:

curl -i 127.0.0.1/.well-known/acme-challenge/test
curl -i 127.0.0.1:54321/.well-known/acme-challenge/test

If the first command produces a 503 but the second one works, then in yet another terminal you can try something like:

ss -tlnp | grep "54321"
tcpdump -i lo 'port 54321'

and while tcpdump is running, try the first curl command again.

Try and narrow down exactly where the breakdown is happening.

2 Likes
13

Right, finally figured this out, and now everything is working AOK.

It turns out that my Configs were correct - and that Certbot does not / can not / will not accept the .acme-well-know via the HAProxy redirect when issuing NEW Certificates, but is perfectly happy to do so on renewing certificates.

Solution:

  1. Turn off the haproxy.service: systemctl stop haproxy
  2. Issue the command: certbot certonly --standalone -d www.example.com
  3. Issue the command: cat /etc/letsencrypt/live/www.example.com/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/private/www.example.com.pem
  4. Turn on the haproxy.service: systemctl start haproxy

Thanks for the help, everybody :slight_smile:

1 Like
14

That would be a bug in certbot? Sounds rather hard to believe actually. Renewing is actually just like getting a new certificate, except for the infornation pulled out of a configuration file and exsisting cert. The challenge handeling should be the same.

1 Like
15

Hard to believe or not, to quote The Great Detective: “How often have I said that when you have excluded the impossible whatever remains, however improbable, must be the truth.”

I’ve spent days trying to get this to work, and my previous post is the only way I found to get it done.

(Yes, once I had it working I stopped looking - obviously - but there’s nothing else I’ve been able to find online or going through both the HAProxy Logs and the Certbot Logs that tells me any different. I’ll happily be proven wrong - but, to quote my favourite animated character: “I don’t think so.”) :slight_smile:

16

Well, of course it’s great this works for you, but to me it sounds like a workaround, bypassing the problem, not fixing it.

You say you’ve excluded the impossible. We’ll only have to take your word for it, because you haven’t shared any debugging info like @_az suggested. Perhaps there is something there you might have overlooked. Not because your skills aren’t good enough, but two see more than one.

1 Like
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.