Certbot-auto fails / worked just fine until Dec '19

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: bereskapi-ha.duckdns.org

I ran this command: ./certbot-auto certonly -vvv --standalone --preferred-challenges http-01 --email bereskapi@gmail.com -d bereskapi-ha.duckdns.org

It produced this output:
Requesting to rerun ./certbot-auto with root privileges…
./certbot-auto has insecure permissions!
To learn how to fix them, visit Certbot-auto deployment best practices
Root logging level set at -10
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator standalone and installer None
Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator
Initialized: <certbot._internal.plugins.standalone.Authenticator object at 0x75979d10>
Prep: True
Selected authenticator <certbot._internal.plugins.standalone.Authenticator object at 0x75979d10> and installer None
Plugins selected: Authenticator standalone, Installer None
Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u’https://acme-v02.api.letsencrypt.org/acme/acct/44402660’, new_authzr_uri=None, terms_of_service=None), 56e254422c558155b2484ad91dfd16bb, Meta(creation_host=u’bereskapi-ha’, creation_dt=datetime.datetime(2018, 10, 24, 5, 26, 42, tzinfo=)))>
Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
https://acme-v02.api.letsencrypt.org:443 “GET /directory HTTP/1.1” 200 658
Received response:
HTTP 200
Server: nginx
Date: Tue, 11 Feb 2020 10:46:09 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
“keyChange”: “https://acme-v02.api.letsencrypt.org/acme/key-change”,
“meta”: {
“caaIdentities”: [
“letsencrypt.org”
],
“termsOfService”: “https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf”,
“website”: “https://letsencrypt.org”
},
“newAccount”: “https://acme-v02.api.letsencrypt.org/acme/new-acct”,
“newNonce”: “https://acme-v02.api.letsencrypt.org/acme/new-nonce”,
“newOrder”: “https://acme-v02.api.letsencrypt.org/acme/new-order”,
“revokeCert”: “https://acme-v02.api.letsencrypt.org/acme/revoke-cert”,
“xsmBB0P3OUc”: “Adding random entries to the directory”
}
Renewal conf file /etc/letsencrypt/renewal/btcpay.bereskapi-ha.duckdns.org.conf is broken. Skipping.
Traceback was:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/cert_manager.py”, line 381, in _search_lineages
candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/storage.py”, line 465, in init
self._check_symlinks()
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/storage.py”, line 532, in _check_symlinks
“expected {0} to be a symlink”.format(link))
CertStorageError: expected /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org/cert.pem to be a symlink

Should renew, less than 30 days before certificate expiry 2020-02-28 20:12:31 UTC.
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Generating key (2048 bits): /etc/letsencrypt/keys/0082_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0082_csr-certbot.pem
Requesting fresh nonce
Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
https://acme-v02.api.letsencrypt.org:443 “HEAD /acme/new-nonce HTTP/1.1” 200 0
Received response:
HTTP 200
Server: nginx
Date: Tue, 11 Feb 2020 10:46:12 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0001XGbgOLXZs8oI74GbgR6urnjMS9Mx5nLWyyddJQGWR9I
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

Storing nonce: 0001XGbgOLXZs8oI74GbgR6urnjMS9Mx5nLWyyddJQGWR9I
JWS payload:
{
“identifiers”: [
{
“type”: “dns”,
“value”: “bereskapi-ha.duckdns.org”
}
]
}
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
“protected”: “eyJub25jZSI6ICIwMDAxWEdiZ09MWFpzOG9JNzRHYmdSNnVybmpNUzlNeDVuTFd5eWRkSlFHV1I5SSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIiwgImtpZCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzQ0NDAyNjYwIiwgImFsZyI6ICJSUzI1NiJ9”,
“payload”: “ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwgCiAgICAgICJ2YWx1ZSI6ICJiZXJlc2thcGktaGEuZHVja2Rucy5vcmciCiAgICB9CiAgXQp9”,
“signature”: “bddWkBQmJzTod8bL_OJVOXIdl0bhG-tCxeivkKURKaWNlHEXoK-YGKbmNHpMK4iTvxcoajDY0fAmwZCbs0CITrRo8So48Cc-TlKR2bdGJdAVk8YF_sk4zlEm2TwetcoCQJSLIqVAjmpyVRY9IGp3S87qC6OOBDJjZN7lOqEicJPa-eXkD4cfpTN9pT-wL1Otkni7TFV5Kxu-sGp_syipbJItg7rAr1HG63qU5tkM2s71u0lQiX0oBPJ–WVQSn5WPudcLfusdCf06ZkfdrGurn5KaV_psg_cWuaiohQU3nrvTlX249yr_m3n29q1gFvcRV4Fc68oo9iLI0GlF_XQIA”
}
https://acme-v02.api.letsencrypt.org:443 “POST /acme/new-order HTTP/1.1” 201 354
Received response:
HTTP 201
Server: nginx
Date: Tue, 11 Feb 2020 10:46:13 GMT
Content-Type: application/json
Content-Length: 354
Connection: keep-alive
Boulder-Requester: 44402660
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Location: https://acme-v02.api.letsencrypt.org/acme/order/44402660/2298987627
Replay-Nonce: 00027uKEvQ-pUE6HlqWueQmmR_AFcmvXjzfsLNItInRpWug
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
“status”: “pending”,
“expires”: “2020-02-18T10:46:13.156600264Z”,
“identifiers”: [
{
“type”: “dns”,
“value”: “bereskapi-ha.duckdns.org”
}
],
“authorizations”: [
“https://acme-v02.api.letsencrypt.org/acme/authz-v3/2758958011”
],
“finalize”: “https://acme-v02.api.letsencrypt.org/acme/finalize/44402660/2298987627”
}
Storing nonce: 00027uKEvQ-pUE6HlqWueQmmR_AFcmvXjzfsLNItInRpWug
JWS payload:

Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/2758958011:
{
“protected”: “eyJub25jZSI6ICIwMDAyN3VLRXZRLXBVRTZIbHFXdWVRbW1SX0FGY212WGp6ZnNMTkl0SW5ScFd1ZyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjc1ODk1ODAxMSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC80NDQwMjY2MCIsICJhbGciOiAiUlMyNTYifQ”,
“payload”: “”,
“signature”: “urxvbGiqj2f4dnO9XlafuDkeb-IfbzUdayYRVZI-CR5lN_yp7fs0N5xVRArDWIcQdbTXLvRhzfPJ4WvG_CE8qBMJ9PtHr3TH7Wg79uLgkVkNKzKQmP_cSlnTFrj0ir1qoEazzX0R7E89TZdcHhSKSyKHCIDMQ-NaLPEOHxMux3ibBUCbL4UFxy64l8p_-Gj4VqCXxZ7AMUOtKm_cK7W0Q5uLr-inA8XpjR06kd1skENZfX9j7pfRh08Pac-8i8DvrlBemT7YLOrp0SyXIuJv_Mucw0dUOOVB3YdHwiTvtWN120TNY3qjwQUm2_s5ko9atr_Nfg1dhis5-tNUpbHyFw”
}
https://acme-v02.api.letsencrypt.org:443 “POST /acme/authz-v3/2758958011 HTTP/1.1” 200 802
Received response:
HTTP 200
Server: nginx
Date: Tue, 11 Feb 2020 10:46:13 GMT
Content-Type: application/json
Content-Length: 802
Connection: keep-alive
Boulder-Requester: 44402660
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0002SpjCY9dDbClV7GNK8VWpnDCfvk88DxCxb1xRH_xd7XA
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
“identifier”: {
“type”: “dns”,
“value”: “bereskapi-ha.duckdns.org”
},
“status”: “pending”,
“expires”: “2020-02-18T10:46:13Z”,
“challenges”: [
{
“type”: “http-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2758958011/lVqYKQ”,
“token”: “GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k”
},
{
“type”: “dns-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2758958011/K841uA”,
“token”: “GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k”
},
{
“type”: “tls-alpn-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2758958011/1MqW4Q”,
“token”: “GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k”
}
]
}
Storing nonce: 0002SpjCY9dDbClV7GNK8VWpnDCfvk88DxCxb1xRH_xd7XA
Performing the following challenges:
http-01 challenge for bereskapi-ha.duckdns.org
Successfully bound to :80 using IPv6
Certbot wasn’t able to bind to :80 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
Waiting for verification…
JWS payload:
{
“type”: “http-01”,
“resource”: “challenge”
}
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/2758958011/lVqYKQ:
{
“protected”: “eyJub25jZSI6ICIwMDAyU3BqQ1k5ZERiQ2xWN0dOSzhWV3BuRENmdms4OER4Q3hiMXhSSF94ZDdYQSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwtdjMvMjc1ODk1ODAxMS9sVnFZS1EiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNDQ0MDI2NjAiLCAiYWxnIjogIlJTMjU2In0”,
“payload”: “ewogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9”,
“signature”: “OuTeKV5WiEmYUI2Toyugx29254dk2JJXp3UPu2rXU5r14ZZ1pbkHRAgSlTQYkbqZ0khuh0jX7nQjuIk8mrP60ZWnslRa4aT1BEcopaWXkoQrLnDslrWdH8_675tadG4zH-mU1C6Ej2NR_cSx8mbM5hr7NPDpPuW9cPDJjpRNVHoJ_2SBXW_npLPbVrfkvSHhbgPRDqw1mHIBnn2nsVrmUYoLRMs2-l5Pozl4h0jciRvjPTu3Vf0fVRFoD0KplBNSMJ_k75P0D5krpA3qaRenpG-nCyI1JpTpXzYNS7lX9Bv9V88vonA_vD5wScX3gQnbPe2KKYQllmBTX0EvcP480g”
}
https://acme-v02.api.letsencrypt.org:443 “POST /acme/chall-v3/2758958011/lVqYKQ HTTP/1.1” 200 185
Received response:
HTTP 200
Server: nginx
Date: Tue, 11 Feb 2020 10:46:13 GMT
Content-Type: application/json
Content-Length: 185
Connection: keep-alive
Boulder-Requester: 44402660
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”, https://acme-v02.api.letsencrypt.org/acme/authz-v3/2758958011;rel=“up”
Location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/2758958011/lVqYKQ
Replay-Nonce: 0001P59DSEZ9qrr1wcDUtkL1sglbIauwJC3sVdn2Jzh7-Yw
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
“type”: “http-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2758958011/lVqYKQ”,
“token”: “GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k”
}
Storing nonce: 0001P59DSEZ9qrr1wcDUtkL1sglbIauwJC3sVdn2Jzh7-Yw
JWS payload:

Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/2758958011:
{
“protected”: “eyJub25jZSI6ICIwMDAxUDU5RFNFWjlxcnIxd2NEVXRrTDFzZ2xiSWF1d0pDM3NWZG4ySnpoNy1ZdyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjc1ODk1ODAxMSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC80NDQwMjY2MCIsICJhbGciOiAiUlMyNTYifQ”,
“payload”: “”,
“signature”: “lDcv5WjHjhOMyuzDdBPWwPeJozaoAjl2P-v4XKrSXqJYB5_uFhcVih1NM_wBW1IGbQe10rUTM0amu6cnPlrzVB1BNsqDtRVZiS5pPbZysrgUFqIN-2cJpkYwVAtNgOiEnq63f32p0F2N6J8n4HSbXpKmrj8VLOTWMXpBaoUuFMAfFyCPxrCX5Tggc_MOuYb8mkay5h_Uhz30HXGc5Df6wx5CwpysY78YhssZmZfol-p5LFKkmIA5yVqOpXOxJtpwO7Vwg6EfGUkl7fo3_5u28WQ4tEGfO25YYxRnugovDuVB9KR1SWLXTQ63P7elemjen96_l2uZKUVtfftMjUvPXA”
}
https://acme-v02.api.letsencrypt.org:443 “POST /acme/authz-v3/2758958011 HTTP/1.1” 200 1192
Received response:
HTTP 200
Server: nginx
Date: Tue, 11 Feb 2020 10:46:14 GMT
Content-Type: application/json
Content-Length: 1192
Connection: keep-alive
Boulder-Requester: 44402660
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0002Kp6qGWSZQ04-G6IbrFomRuNR6l_N_wfmh4P-hZDapLs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
“identifier”: {
“type”: “dns”,
“value”: “bereskapi-ha.duckdns.org”
},
“status”: “invalid”,
“expires”: “2020-02-18T10:46:13Z”,
“challenges”: [
{
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Invalid response from http://bereskapi-ha.duckdns.org/.well-known/acme-challenge/GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k [193.105.59.205]: “\u003c?xml version=\“1.0\” encoding=\“iso-8859-1\”?\u003e\n\u003c!DOCTYPE html PUBLIC \”-//W3C//DTD XHTML 1.0 Transitional//EN\”\n \“http://www.”",
“status”: 403
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2758958011/lVqYKQ”,
“token”: “GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k”,
“validationRecord”: [
{
“url”: “http://bereskapi-ha.duckdns.org/.well-known/acme-challenge/GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k”,
“hostname”: “bereskapi-ha.duckdns.org”,
“port”: “80”,
“addressesResolved”: [
“193.105.59.205”
],
“addressUsed”: “193.105.59.205”
}
]
}
]
}
Storing nonce: 0002Kp6qGWSZQ04-G6IbrFomRuNR6l_N_wfmh4P-hZDapLs
Challenge failed for domain bereskapi-ha.duckdns.org
http-01 challenge for bereskapi-ha.duckdns.org
Reporting to user: The following errors were reported by the server:

Domain: bereskapi-ha.duckdns.org
Type: unauthorized
Detail: Invalid response from http://bereskapi-ha.duckdns.org/.well-known/acme-challenge/GUIwdexIgpdIkw5wcOssWYtGw2O5Pv-UBDNhctAnZ5k [193.105.59.205]: "<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN”\n “http://www.”

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
Encountered exception:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/auth_handler.py”, line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/auth_handler.py”, line 180, in _poll_authorizations
raise errors.AuthorizationError(‘Some challenges have failed.’)
AuthorizationError: Some challenges have failed.

Calling registered functions
Cleaning up challenges
Stopping server at :::80…
Exiting abnormally:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 15, in main
return internal_main.main(cli_args)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/main.py”, line 1347, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/main.py”, line 1233, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/main.py”, line 116, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/renewal.py”, line 306, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/client.py”, line 344, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/client.py”, line 391, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/auth_handler.py”, line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/auth_handler.py”, line 180, in _poll_authorizations
raise errors.AuthorizationError(‘Some challenges have failed.’)
AuthorizationError: Some challenges have failed.
Some challenges have failed.

IMPORTANT NOTES:

My web server is (include version): nginx in organizr docker container

The operating system my web server runs on is (include version): Raspbian GNU/Linux 9.11 (stretch)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 1.2.0

1 Like
2

Make sure your IP is correct/updated.
Logs (and DNS) show: 193.105.59.205

curl -Iki http://bereskapi-ha.duckdns.org/
HTTP/1.1 404 Not Found
Content-Type: text/html
Content-Length: 345
X-Frame-Options: SAMEORIGIN
Date: Tue, 11 Feb 2020 17:43:45 GMT
Server: lighttpd/1.4.39

curl -Iki http://193.105.59.205/
HTTP/1.1 404 Not Found
Content-Type: text/html
Content-Length: 345
X-Frame-Options: SAMEORIGIN
Date: Tue, 11 Feb 2020 17:44:10 GMT
Server: lighttpd/1.4.39

If the IP is correct, check the webserver configs.

1 Like
3

Possibly related to what @rg305 noticed, you can’t use --standalone while you have another web server listening on port 80. You can use it if you stop the other server temporarily during the renewal process. Certbot can do this automatically with --pre-hook and --post-hook options if you want.

2 Likes
4

thank you
ip is correct and so is the duckdns token for this domain
I run nginx reverse proxy inside organizr docker container listening on port 80
I always stop this container before running certbot-auto renew
And I have bash script to automate this renewal every month
Everything worked until end of last year
here is my nginx config:
pi@bereskapi-ha:~ $ cat /opt/organizr/nginx/site-confs/default

Redirect all http traffic to https

server {
listen 80;
return 301 https://$host$request_uri;
}

Portainer Upstream

upstream portainer {
server 192.168.2.230:9000;
keepalive 32;
}

Home Assistant Upstream

upstream homeassistant {
server 192.168.2.230:8123;
keepalive 32;
}

Node-RED Upstream

upstream nodered {
server 192.168.2.230:1880;
keepalive 32;
}

Organizr Upstream

upstream organizr {
server 192.168.2.230:443;
keepalive 32;
}

server {
listen 443 ssl http2;

    root /config/www;
    index index.html index.htm index.php;

    server_name portainer.bereskapi-ha.duckdns.org;

    client_max_body_size 0;

    ssl_certificate /etc/letsencrypt/live/portainer.bereskapi-ha.duckdns.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/portainer.bereskapi-ha.duckdns.org/privkey.pem;
    ssl_protocols TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    ssl_prefer_server_ciphers on;

    location / {
            proxy_pass http://portainer/;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
    }

    location /api/websocket/ {
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_http_version 1.1;
            proxy_pass http://portainer/api/websocket/;
    }

}

server {
listen 443 ssl http2;

    root /config/www;
    index index.html index.htm index.php;

    server_name ha.bereskapi-ha.duckdns.org;

    client_max_body_size 0;

    ssl_certificate /etc/letsencrypt/live/ha.bereskapi-ha.duckdns.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ha.bereskapi-ha.duckdns.org/privkey.pem;
    ssl_protocols TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    ssl_prefer_server_ciphers on;

    location / {
            proxy_pass http://homeassistant/;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
    }

location /api/websocket {
        proxy_pass http://homeassistant/api/websocket;
        proxy_set_header Host $host;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

}

server {
listen 443 ssl http2;

    root /config/www;
    index index.html index.htm index.php;

    server_name nodered.bereskapi-ha.duckdns.org;

    client_max_body_size 0;

    ssl_certificate /etc/letsencrypt/live/nodered.bereskapi-ha.duckdns.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/nodered.bereskapi-ha.duckdns.org/privkey.pem;
    ssl_protocols TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://nodered/;
        proxy_http_version 1.1;
        proxy_set_header Connection "upgrade";
        proxy_set_header Upgrade $http_upgrade;
    }

}

server {
listen 443 ssl http2 default_server;
server_name bereskapi-ha.duckdns.org;
root /config/www/Dashboard;
index index.html index.htm index.php;

client_max_body_size 0;

#SSL settings
ssl_certificate /etc/letsencrypt/live/bereskapi-ha.duckdns.org/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/bereskapi-ha.duckdns.org/privkey.pem;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
ssl_prefer_server_ciphers on;

location / {
    try_files $uri $uri/ /index.html /index.php?$args =404;
}

location ~ \.php$ {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    # With php5-cgi alone:
    fastcgi_pass 127.0.0.1:9000;
    # With php5-fpm:
    #fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
    include /etc/nginx/fastcgi_params;
}

# BLOCK ORGANIZR DASHBOARD FILES
location ~ /loginLog.json|chat.db|users.db|org.log {
    return 404;
}

location /auth-admin {
    internal;
    proxy_pass http://organizr/auth.php?admin;
    proxy_set_header Content-Length "";
}

location /auth-user {
    internal;
    proxy_pass http://organizr/auth.php?user;
    proxy_set_header Content-Length "";
}

}

#SSL termination and offload for btcpay

#upstream btcpayserver {

server 192.168.2.135:8083;

server 192.168.2.135:4430;

#}

#server {

listen 443 ssl http2;

server_name btcpay.bereskapi-ha.duckdns.org;

ssl_certificate /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org/fullchain.pem;

ssl_certificate_key /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org/privkey.pem;

ssl_protocols TLSv1.1 TLSv1.2;

ssl_ciphers “EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4”;

add_header Strict-Transport-Security “max-age=31536000; includeSubdomains”;

ssl_prefer_server_ciphers on;

location / {

proxy_pass http://btcpayserver;

proxy_http_version 1.1;

proxy_set_header Connection “upgrade”;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forwarded-Proto https;

}

#}

#SSL termination and offload for picamera

upstream camera {
server 192.168.2.135:8080;
}

server {
listen 443 ssl http2;
server_name picamera.bereskapi-ha.duckdns.org;

ssl_certificate /etc/letsencrypt/live/picamera.bereskapi-ha.duckdns.org/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/picamera.bereskapi-ha.duckdns.org/privkey.pem;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
ssl_prefer_server_ciphers on;

location / {
    proxy_pass http://camera;
    proxy_http_version 1.1;
    proxy_set_header Connection "upgrade";
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
}

}

5

and here is the nginx.conf:

pi@bereskapi-ha:~ $ cat /opt/organizr/nginx/nginx.conf

Version 2018/08/16 - Changelog: https://github.com/linuxserver/docker-baseimage-nginx-armhf/commits/master/root/defaults/nginx.conf

user abc;
worker_processes 4;
pid /run/nginx.pid;
include /etc/nginx/modules/*.conf;

events {
worker_connections 768;
# multi_accept on;
}

http {

##
# Basic Settings
##

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;

server_names_hash_bucket_size 64;
# server_name_in_redirect off;

client_max_body_size 0;

include /etc/nginx/mime.types;
default_type application/octet-stream;

##
# Logging Settings
##

access_log /config/log/nginx/access.log;
error_log /config/log/nginx/error.log;

##
# Gzip Settings
##

gzip on;
gzip_disable "msie6";

# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

##
# nginx-naxsi config
##
# Uncomment it if you installed nginx-naxsi
##

#include /etc/nginx/naxsi_core.rules;

##
# nginx-passenger config
##
# Uncomment it if you installed nginx-passenger
##

#passenger_root /usr;
#passenger_ruby /usr/bin/ruby;

##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /config/nginx/site-confs/*;

}

#mail {

# See sample authentication script at:

# http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript

# auth_http localhost/auth.php;

# pop3_capabilities “TOP” “USER”;

# imap_capabilities “IMAP4rev1” “UIDPLUS”;

server {

listen localhost:110;

protocol pop3;

proxy on;

}

server {

listen localhost:143;

protocol imap;

proxy on;

}

#}
daemon off;

6

but it should not matter actually because docker container with nginx is stopped when certbot renews certs

7

"<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN”\n “http://www.”

This is the same HTML page that's visible there now, so apparently the nginx process is not being successfully stopped during the renewal.

1 Like
8

thank you
right now nginx is running because organizr container is up
but if I stop it than no nginx is running:
pi@bereskapi-ha:~ ps aux | grep nginx root 13535 0.0 0.0 196 4 ? S фев11 0:00 s6-supervise nginx root 13541 0.0 0.3 4352 2956 ? Ss фев11 0:00 nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf pi 13558 0.0 0.3 4968 3148 ? S фев11 0:05 nginx: worker process pi 13559 0.0 0.2 4876 2480 ? S фев11 0:00 nginx: worker process pi 13560 0.0 0.1 4536 956 ? S фев11 0:00 nginx: worker process pi 13561 0.0 0.1 4536 956 ? S фев11 0:00 nginx: worker process pi 27528 0.0 0.0 5992 548 pts/0 S+ 00:24 0:00 grep --color=auto nginx pi@bereskapi-ha:~ docker stop organizr
organizr
pi@bereskapi-ha:~ $ ps aux | grep nginx
pi 27956 0.0 0.0 5992 524 pts/0 S+ 00:25 0:00 grep --color=auto nginx

9

Nonetheless, the Let’s Encrypt service reported in the error message that it got that particular HTML data back from your server when it tried to connect. So the method that was meant to stop the container during the renewal apparently didn’t have the intended effect.

1 Like
10

then I am lost(
no nginx process is running after stopping organizr
what else could be running that messes up with certbot?!

11

Can you try running curl bereskapi-ha.duckdns.org after stopping it? (Ideally from outside your network, if possible.)

1 Like
12

MacBook-Air-DG-4007:~ bereska$ curl bereskapi-ha.duckdns.org

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

13

MacBook-Air-DG-4007:~ bereska$ curl bereskapi-ha.duckdns.org

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

14

MacBook-Air-DG-4007:~ bereska$ curl bereskapi-ha.duckdns.org

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

MacBook-Air-DG-4007:~ bereska$
15

404 is a response from a web server.
ping bereskapi-ha.duckdns.org
[what IP does it try?]

1 Like
16

Yes, these results mean that there is a web server which is still listening for requests there. It might be inside the container or outside the container, but it’s occupying port 80 on the server!

One possibility is that you have an nginx instance outside the container that’s configured to forward incoming HTTP requests to different locations (such as to specific containers). If so, that hypothetical instance isn’t configured properly for the way that you’re using Certbot because it isn’t forwarding the certificate authority’s challenge requests to the Certbot container appropriately.

2 Likes
17

pi@bereskapi-ha:/opt sudo lsof -i:80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME docker-pr 4726 root 4u IPv6 341732880 0t0 TCP *:http (LISTEN) pi@bereskapi-ha:/opt docker stop organizr
organizr
pi@bereskapi-ha:/opt sudo lsof -i:80 pi@bereskapi-ha:/opt

MacBook-Air-DG-4007:~ bereska$ ping bereskapi-ha.duckdns.org

PING bereskapi-ha.duckdns.org (193.105.59.205): 56 data bytes

64 bytes from 193.105.59.205: icmp_seq=0 ttl=28 time=297.567 ms

18

nothing is running on port 80 after stopping organizr
the ip is correct

19

404 response from curl bereskapi-ha.duckdns.org WITH or WITHOUT organizr running
the response from curl https://bereskapi-ha.duckdns.org WITH organizr running is as expected html page

20

So you can connect to your web server via the external(Internet) IP?
[that would require hairpinning on the gateway - if you are you on the same local subnet]

Please show output of:
curl -Iki http://bereskapi-ha.duckdns.org/

(on port 80) I get:
curl -Iki http://bereskapi-ha.duckdns.org/
HTTP/1.1 404 Not Found
Content-Type: text/html
Content-Length: 345
X-Frame-Options: SAMEORIGIN
Date: Wed, 12 Feb 2020 00:15:49 GMT
Server: lighttpd/1.4.39

[NOT NGINX]

While (on port 443), I get:
curl -Iki https://bereskapi-ha.duckdns.org/
HTTP/2 200
server: nginx/1.14.0
date: Wed, 12 Feb 2020 00:17:48 GMT
content-type: text/html; charset=UTF-8
x-powered-by: PHP/7.2.10
set-cookie: PHPSESSID=4iegqc3ae42msfhusjka8hcg3c; path=/
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate
pragma: no-cache
set-cookie: cookiePassword=; expires=Tue, 11-Feb-2020 00:17:48 GMT; path=/; domain=.duckdns.org; secure; HttpOnly
set-cookie: Auth=; expires=Tue, 11-Feb-2020 00:17:48 GMT; path=/; domain=.duckdns.org; secure; HttpOnly
set-cookie: mpt=; expires=Tue, 11-Feb-2020 00:17:48 GMT; path=/; domain=.duckdns.org; secure; HttpOnly
set-cookie: Organizr_Token=; expires=Tue, 11-Feb-2020 00:17:48 GMT; path=/; domain=.duckdns.org; secure; HttpOnly
strict-transport-security: max-age=31536000; includeSubdomains

1 Like