Certbot --dry-run succeeds, but "real" verification fails

First, I’m not much of a server admin. It’s one of the things I struggle with the most. So perhaps I’m missing something obvious. But, I’m trying to set up a jenkins server at jenkins.ucdev.net. The server uses

  • RHEL 7
  • apache 2.4.6
  • certbot v0.36.0
  • I have root access
  • the server is (in general) privately managed by my IT dept

Jenkins has a webroot at /var/cache/jenkins/war in its config (/etc/default/jenkins) .

<VirtualHost *:80>
	Header set X-Robots-Tag: none

    DocumentRoot /var/cache/jenkins/war        

	ServerName jenkins.ucdev.net
	ServerAlias jenkins.ucdev.net

	ProxyPass / http://127.0.0.1:8080/
	ProxyPassReverse / http://127.0.0.1:8080/
	ProxyRequests Off
	AllowEncodedSlashes NoDecode
</VirtualHost>

With al this, I can access the jenkins application at the url behind my CAS plugin.

When I use the certbot certonly --dry-run, the certification succeeds. However, when I then do exactly the same commands and parameters, the certification fails.

The with verbose logging, the error I get back is

2019-08-21 14:16:21,882:DEBUG:acme.client:Received response:
HTTP 200
content-length: 1712
expires: Wed, 21 Aug 2019 18:16:21 GMT
cache-control: max-age=0, no-cache, no-store
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
pragma: no-cache
boulder-requester: 40668949
date: Wed, 21 Aug 2019 18:16:21 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0001rcUDEfLn2gjlSQhNFVHsTHpViiPXWISJvovQgsXhQRw

{
  "identifier": {
    "type": "dns",
    "value": "jenkins.ucdev.net"
  },
  "status": "invalid",
  "expires": "2019-08-28T18:16:05Z",
  "challenges": [
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/oPwj_ZhYeBr-aEr8xDL5ifeAzUaE7dHRVYkl0aj1SpQ/19836695963",
      "token": "bh_0Tc-5n_NxBBJe0_lDnGkKoAN7ZBnXqafEzy09ZaM"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://jenkins.ucdev.net/.well-known/acme-challenge/X2XGpJcSfE5gWY1OmFPZLmryqH12ZS4gVB1UDrH288o [137.99.51.36]: \"\u003chtml\u003e\u003chead\u003e\u003cmeta http-equiv='refresh' content='1;url=/securityRealm/commenceLogin?from=%!F(MISSING).well-known%!F(MISSING)acme-challenge%!F(MISSING)X2XGpJ\"",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/oPwj_ZhYeBr-aEr8xDL5ifeAzUaE7dHRVYkl0aj1SpQ/19836695964",
      "token": "X2XGpJcSfE5gWY1OmFPZLmryqH12ZS4gVB1UDrH288o",
      "validationRecord": [
        {
          "url": "http://jenkins.ucdev.net/.well-known/acme-challenge/X2XGpJcSfE5gWY1OmFPZLmryqH12ZS4gVB1UDrH288o",
          "hostname": "jenkins.ucdev.net",
          "port": "80",
          "addressesResolved": [
            "137.99.51.36"
          ],
          "addressUsed": "137.99.51.36"
        }
      ]
    },
    {
      "type": "dns-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/oPwj_ZhYeBr-aEr8xDL5ifeAzUaE7dHRVYkl0aj1SpQ/19836695965",
      "token": "CzCr1gW_tPvbEws0U-BrCKZxtFUX-GA9xUbdL_-bV98"
    }
  ]
}
2019-08-21 14:16:21,882:DEBUG:acme.client:Storing nonce: 0001rcUDEfLn2gjlSQhNFVHsTHpViiPXWISJvovQgsXhQRw
2019-08-21 14:16:21,884:WARNING:certbot.auth_handler:Challenge failed for domain jenkins.ucdev.net
2019-08-21 14:16:21,884:INFO:certbot.auth_handler:http-01 challenge for jenkins.ucdev.net
2019-08-21 14:16:21,885:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: jenkins.ucdev.net
Type:   unauthorized
Detail: Invalid response from http://jenkins.ucdev.net/.well-known/acme-challenge/X2XGpJcSfE5gWY1OmFPZLmryqH12ZS4gVB1UDrH288o [137.99.51.36]: "<html><head><meta http-equiv='refresh' content='1;url=/securityRealm/commenceLogin?from=%!F(MISSING).well-known%!F(MISSING)acme-challenge%!F(MISSING)X2XGpJ"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2019-08-21 14:16:21,886:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 154, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

2019-08-21 14:16:21,886:DEBUG:certbot.error_handler:Calling registered functions
2019-08-21 14:16:21,886:INFO:certbot.auth_handler:Cleaning up challenges
2019-08-21 14:16:21,886:DEBUG:certbot.plugins.webroot:Removing /var/cache/jenkins/war/.well-known/acme-challenge/X2XGpJcSfE5gWY1OmFPZLmryqH12ZS4gVB1UDrH288o
2019-08-21 14:16:21,887:DEBUG:certbot.plugins.webroot:All challenges cleaned up
2019-08-21 14:16:21,887:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.36.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1381, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1264, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 120, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 406, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 349, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 385, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 154, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

Any advice or help you could offer would be amazing!

Hi @aberkow

I don't know if this is the problem.

But it's always wrong to have a ServerAlias with the same value as the ServerName.

So remove the ServerAlias row.

ServerAlias -> other domain name, not the same as ServerName.

PS: If this is changed, what says

apachectl -S

hi @JuergenAuer!

thanks for writing back so quickly. I

  • commented out the ServerAlias
  • restarted apache systemctl restart httpd
  • checked httpd -S (below)

unfortunately it’s the same result. My colleague just reminded me that there may be an issue because we’re using SE Linux on this server (IT’s requirement) and we’ve had permissions problems in the past. Could that be part of the issue?

log follows

VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server advance.uconn.edu (/etc/httpd/sites-enabled/advance.uconn.edu.conf:2)
         port 80 namevhost advance.uconn.edu (/etc/httpd/sites-enabled/advance.uconn.edu.conf:2)
                 alias advance.uconn.edu
         port 80 namevhost inauguration.uconn.edu (/etc/httpd/sites-enabled/inauguration.uconn.edu.conf:2)
                 alias inauguration.uconn.edu
         port 80 namevhost jenkins.ucdev.net (/etc/httpd/sites-enabled/jenkins.ucdev.net.conf:2)
         port 80 namevhost news.uconn.edu (/etc/httpd/sites-enabled/news.uconn.edu.conf:1)
                 alias news.uconn.edu
         port 80 namevhost packages.ucdev.net (/etc/httpd/sites-enabled/packages.ucdev.net.conf:2)
                 alias packages.ucdev.net
         port 80 namevhost staging.ucdev.net (/etc/httpd/sites-enabled/staging.ucdev.net.conf:9)
                 alias staging.ucdev.net
                 wild alias *.staging.ucdev.net
*:443                  is a NameVirtualHost
         default server communicationsproto.uconn.edu (/etc/httpd/conf.d/ssl.conf:56)
         port 443 namevhost communicationsproto.uconn.edu (/etc/httpd/conf.d/ssl.conf:56)
         port 443 namevhost packages.ucdev.net (/etc/httpd/sites-available/packages.ucdev.net-le-ssl.conf:2)
                 alias packages.ucdev.net
ServerRoot: "/etc/httpd"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/etc/httpd/logs/error_log"
Mutex default: dir="/run/httpd/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex authdigest-client: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
PidFile: "/run/httpd/httpd.pid"
Define: _RH_HAS_HTTPPROTOCOLOPTIONS
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="apache" id=48
Group: name="apache" id=48

Your configuration looks ok.

But checking your domain - there is no Apache - https://check-your-website.server-daten.de/?q=jenkins.ucdev.net

Domainname Http-Status redirect Sec. G
http://jenkins.ucdev.net/
137.99.51.36 403 0.223 M
Forbidden
https://jenkins.ucdev.net/
137.99.51.36 200 3.657 N
Certificate error: RemoteCertificateNameMismatch, RemoteCertificateChainErrors
http://jenkins.ucdev.net/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
137.99.51.36 403 0.223 M
Forbidden
Visible Content: Authentication required
Info: Html-Content with meta and/or script, may be a problem creating a Letsencrypt certificate using http-01 validation

http has a

Server: Jetty(9.4.z-SNAPSHOT)

And /.well-known/acme-challenge/random-filename doesn't answer with the expected http status 404 - Not Found.

Instead, there is a blocking 403 and a http refresh:

<html><head><meta http-equiv='refresh' content='1;url=/securityRealm/commenceLogin?from=%2F.well-known%2Facme-challenge%2Fcheck-your-website-dot-server-daten-dot-de'/><script>window.location.replace('/securityRealm/commenceLogin?from=%2F.well-known%2Facme-challenge%2Fcheck-your-website-dot-server-daten-dot-de');</script></head><body style='background-color:white; color:white;'> Authentication required <!-- You are authenticated as: anonymous Groups that you are in: Permission you need to have (but didn't): hudson.model.Hudson.Read ... which is implied by: hudson.security.Permission.GenericRead ... which is implied by: hudson.model.Hudson.Administer --> </body></html>

So if you use webroot:

  • Is this the correct webroot of that Jetty?
  • A http status 403 -> can't work, if there is a file, a http status 200 is required.

Seeing your response, and I’m not clear if that’s the webroot for jetty to be honest. As far as I can tell, jetty is the webserver for jenkins, but I can still use apache to get to it.

This jenkins documentation has an interesting thing I’m going to try. Under the mod_proxy heading, it says

Note that this does not apply to the ProxyPassMatch directive, which behaves differently than ProxyPass. Below is an example of ProxyPassMatch to proxy all URLs other than /.well-known (a URL required by letsencrypt):

ProxyPassMatch  ^/(?\!.well-known)  http://localhost:8080 nocanon

Perhaps that’s the part I’m missing (although I need to look up what ProxyPassMatch does…)

UPDATE - The regex in that documentation fails, but I haven’t quite figured out the correct regex yet.

Looks like this is the missing element.

So /.well-known/acme-challenge wouldn't be proxied to the Jetty Server.

Currently, it doesn't work -

http://jenkins.ucdev.net/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de

-> Jetty Server, not the Apache.

First: I don't know if this is actually the right strategy. However, regarding the regular expression syntax...

Looks like the backslash is in the wrong place. It should be before the ".", not before the "!". Like:

ProxyPassMatch  ^/(?!\.well-known)  http://localhost:8080 nocanon

If the Apache configuration syntax doesn't like that, maybe quote it:

ProxyPassMatch  "^/(?!\.well-known)"  http://localhost:8080 nocanon

However, Let's Encrypt only needs the /.well-known/acme-challenge/ path. Not everything matching /.well-known. I'd try:

ProxyPassMatch  "^/(?!\.well-known/acme-challenge/)"  http://localhost:8080 nocanon

But, again, I don't know if this is the correct strategy to solve your problem.

@mnordhoff thanks for that!

Now I’m making progress by getting a different error which usually means I’m on the right track :sweat_smile:. Apache didn’t like using ProxyPassMatch with regex in this context. But it seems to be ok with

ProxyPass "/.well-known/acme-challenge" "!"
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/

However, when I getting the certificates, I get a 404 error at http://jenkins.ucdev.net/.well-known/acme-challenge/ instead of a 403. This seems to be an improvement because of @JuergenAuer’s comment above.

I’m not entirely sure what to try next and I don’t want to run certbot too many times and get locked out. I may try setting a document root for apache that’s unrelated to the jenkins application.

UPDATE - when I curl http://jenkins.ucdev.net/.well-known/acme-challenge/test I get back test test test which is what I entered into the test file. So, this may be an SE Linux problem. Perhaps certbot can’t write to the directory correctly.

Yep, your test file

http://jenkins.ucdev.net/.well-known/acme-challenge/test

works, that's very good.

What's the directory where you have created the subdirectories .well-known/acme-challenge ?

That's the directory you should use with the -w parameter.

Ok this is interesting. The directory being used is actually /var/www/html and not /var/cache/jenkins/war. I think I created that file many errors ago and forgot where I put it. I’m going to try passing /var/www/html as the document root to certbot and see what I get.

UPDATE - That worked! so to sum up, the current state of things is

  • jenkins running in it’s own server
  • apache using it’s own document root /var/www/html
  • apache config follows
<VirtualHost *:80>
	# no indexing
	Header set X-Robots-Tag: none

	ServerName jenkins.ucdev.net

	ProxyPass "/.well-known/acme-challenge" "!"

	ProxyPass / http://127.0.0.1:8080/
	ProxyPassReverse / http://127.0.0.1:8080/

	ProxyRequests Off
	AllowEncodedSlashes NoDecode
</VirtualHost>

Thanks @JuergenAuer and @mnordhoff! I think this solves this issue for now. If/when I have more questions, I’ll open another topic.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.