Lets-encrypt failing

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
genelab-galaxy.usra.edu
I ran this command:
sudo certbot --apache -d genelab-galaxy.usra.edu
It produced this output:

sudo certbot --apache -d genelab-galaxy.usra.edu
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for genelab-galaxy.usra.edu
Waiting for verification...
Challenge failed for domain genelab-galaxy.usra.edu
http-01 challenge for genelab-galaxy.usra.edu
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: genelab-galaxy.usra.edu
   Type:   unauthorized
   Detail: Invalid response from
   http://genelab-galaxy.usra.edu/.well-known/acme-challenge/bpPPHx3VO_1q-XqZplPeVGVXwB7zxyHMN4tkmwzsuE8
   [63.96.143.215]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>403
   Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version):

The operating system my web server runs on is (include version):
Server version: Apache/2.4.6 (CentOS)
My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

certbot 0.38.0

Hi @chawlak11,

I think this is a bug in Certbot, but we haven’t previously identified the cause because people have helpfully provided alternative methods of getting certificates that didn’t involve --apache. However, it would be great for us and other users if you could give us information to help us get to the bottom of this.

Could you post the contents of the log file from /var/log/letsencrypt/letsencrypt.log, and also tell us what version of CentOS you’re running?

Here are the logs

sudo cat letsencrypt/letsencrypt.log
2019-10-18 14:19:36,437:DEBUG:certbot.main:certbot version: 0.38.0
2019-10-18 14:19:36,437:DEBUG:certbot.main:Arguments: ['--apache', '-d', 'genelab-galaxy.usra.edu']
2019-10-18 14:19:36,438:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-10-18 14:19:36,459:DEBUG:certbot.log:Root logging level set at 20
2019-10-18 14:19:36,459:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-10-18 14:19:36,460:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2019-10-18 14:19:36,855:DEBUG:certbot_apache.configurator:Apache version is 2.4.6
2019-10-18 14:19:37,478:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7ff8b21bd490>
Prep: True
2019-10-18 14:19:37,479:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_centos.CentOSConfigurator object at 0x7ff8b21bd490> and installer <certbot_apache.override_centos.CentOSConfigurator object at 0x7ff8b21bd490>
2019-10-18 14:19:37,479:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2019-10-18 14:19:37,526:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u'https://acme-v02.api.letsencrypt.org/acme/acct/69588500', new_authzr_uri=None, terms_of_service=None), d4d24dbb09eef8690556ff06398d3f42, Meta(creation_host=u'galaxy.usra.edu', creation_dt=datetime.datetime(2019, 10, 16, 20, 54, 51, tzinfo=<UTC>)))>
2019-10-18 14:19:37,539:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-10-18 14:19:37,543:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2019-10-18 14:19:37,750:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 658
2019-10-18 14:19:37,751:DEBUG:acme.client:Received response:
HTTP 200
content-length: 658
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
cache-control: public, max-age=0, no-cache
date: Fri, 18 Oct 2019 19:19:37 GMT
x-frame-options: DENY
content-type: application/json

Version

CentOS Linux release 7.7.1908 (Core)
NAME=“CentOS Linux”
VERSION=“7 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“7”

@chawlak11, I think the contents of the log file you provided are incomplete. There should be quite a bit more in the file including the initial error output you provided which would help us take a look at the problem.

If you’ve run Certbot since your initial post, the log output may be in another file like /var/log/letsencrypt/letsencrypt.log.1.

1 Like

well,when i try to upload the entire log file, I get the error message that new users can only posts 20 links.

Is there any better way to share the logs?

2019-10-18 14:35:53,525:DEBUG:certbot.error_handler:Calling registered functions
2019-10-18 14:35:53,525:INFO:certbot.auth_handler:Cleaning up challenges
2019-10-18 14:35:54,058:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.38.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1378, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1133, in run
    certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 121, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 405, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 348, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 384, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 154, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

via paste or drive services.

There is https://pastebin.com/
and then there are…

2 Likes

I've upgraded your account's trust level to "basic user", so that should not be a problem anymore. :slightly_smiling_face:

1 Like

Thanks !!

Here:

sudo cat /var/log/letsencrypt/letsencrypt.log.1
2019-10-18 14:35:47,328:DEBUG:certbot.main:certbot version: 0.38.0
2019-10-18 14:35:47,328:DEBUG:certbot.main:Arguments: ['--apache', '-d', 'genelab-galaxy.usra.edu']
2019-10-18 14:35:47,328:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-10-18 14:35:47,350:DEBUG:certbot.log:Root logging level set at 20
2019-10-18 14:35:47,350:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-10-18 14:35:47,351:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2019-10-18 14:35:47,505:DEBUG:certbot_apache.configurator:Apache version is 2.4.6
2019-10-18 14:35:48,110:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7f1333467490>
Prep: True
2019-10-18 14:35:48,111:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_centos.CentOSConfigurator object at 0x7f1333467490> and installer <certbot_apache.override_centos.CentOSConfigurator object at 0x7f1333467490>
2019-10-18 14:35:48,111:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2019-10-18 14:35:48,139:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u'https://acme-v02.api.letsencrypt.org/acme/acct/69588500', new_authzr_uri=None, terms_of_service=None), d4d24dbb09eef8690556ff06398d3f42, Meta(creation_host=u'galaxy.usra.edu', creation_dt=datetime.datetime(2019, 10, 16, 20, 54, 51, tzinfo=<UTC>)))>
2019-10-18 14:35:48,149:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-10-18 14:35:48,154:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2019-10-18 14:35:48,329:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 658
2019-10-18 14:35:48,330:DEBUG:acme.client:Received response:
HTTP 200
content-length: 658
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
cache-control: public, max-age=0, no-cache
date: Fri, 18 Oct 2019 19:35:48 GMT
x-frame-options: DENY
content-type: application/json

{
  "0CbMTIC0wxo": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2019-10-18 14:35:48,332:INFO:certbot.main:Obtaining a new certificate
2019-10-18 14:35:48,547:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0017_key-certbot.pem
2019-10-18 14:35:48,550:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0017_csr-certbot.pem
2019-10-18 14:35:48,550:DEBUG:acme.client:Requesting fresh nonce
2019-10-18 14:35:48,551:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2019-10-18 14:35:48,592:DEBUG:requests.packages.urllib3.connectionpool:"HEAD /acme/new-nonce HTTP/1.1" 200 0
2019-10-18 14:35:48,593:DEBUG:acme.client:Received response:
HTTP 200
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
cache-control: public, max-age=0, no-cache
date: Fri, 18 Oct 2019 19:35:48 GMT
x-frame-options: DENY
replay-nonce: 0001NF-67-KkXYftX69lW7WNNmOX_R47h_m5RV3QXuDPcw4


2019-10-18 14:35:48,593:DEBUG:acme.client:Storing nonce: 0001NF-67-KkXYftX69lW7WNNmOX_R47h_m5RV3QXuDPcw4
2019-10-18 14:35:48,594:DEBUG:acme.client:JWS payload:
{
  "identifiers": [
    {
      "type": "dns",
      "value": "genelab-galaxy.usra.edu"
    }
  ]
}
2019-10-18 14:35:48,597:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJub25jZSI6ICIwMDAxTkYtNjctS2tYWWZ0WDY5bFc3V05ObU9YX1I0N2hfbTVSVjNRWHVEUGN3NCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIiwgImtpZCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzY5NTg4NTAwIiwgImFsZyI6ICJSUzI1NiJ9",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwgCiAgICAgICJ2YWx1ZSI6ICJnZW5lbGFiLWdhbGF4eS51c3JhLmVkdSIKICAgIH0KICBdCn0",
  "signature": "j35Yzo9Q1A59eSoW9O4r041woXx08szCTxyFvenXJ4cLxHLYjYQuwuWFOVc4G-0KG1nSPK2c9b3S-PrN6vYCl0_xnXYvgaX8tpJpq669nZ_4D5wfU-_Ky2LdZ8fGEIt2tgsaHaMwE0m21FQdqJfVPaP7Cq1mq3yP9oWQkuiSscRWnwXA-xkNv6frh1TiFYQl1I4AvOgfFQkA87Yd_ANHqlAUn6fi1awlOf8fLZ5-7Y74YTtiuSaAnHMK_FY--FzURGZrCxh-jz406Z17cBJLSkJ9mXD8QyU8jvcjleKihkvY6wQHNevZHRiv8ur9950F5DdOR_fQSb0C9moRVuPIbA"
}
2019-10-18 14:35:48,712:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/new-order HTTP/1.1" 201 352
2019-10-18 14:35:48,713:DEBUG:acme.client:Received response:
HTTP 201
content-length: 352
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
location: https://acme-v02.api.letsencrypt.org/acme/order/69588500/1313784331
boulder-requester: 69588500
date: Fri, 18 Oct 2019 19:35:48 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0002iZt4SurjYnldTFGrexLVNZ2KRykpIhyhxd8jqjiOtNI

{
  "status": "pending",
  "expires": "2019-10-25T19:35:48.683194759Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "genelab-galaxy.usra.edu"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/837052177"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/69588500/1313784331"
}
2019-10-18 14:35:48,714:DEBUG:acme.client:Storing nonce: 0002iZt4SurjYnldTFGrexLVNZ2KRykpIhyhxd8jqjiOtNI
2019-10-18 14:35:48,714:DEBUG:acme.client:JWS payload:

2019-10-18 14:35:48,717:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/837052177:
{
  "protected": "eyJub25jZSI6ICIwMDAyaVp0NFN1cmpZbmxkVEZHcmV4TFZOWjJLUnlrcEloeWh4ZDhqcWppT3ROSSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvODM3MDUyMTc3IiwgImtpZCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzY5NTg4NTAwIiwgImFsZyI6ICJSUzI1NiJ9",
  "payload": "",
  "signature": "A3em935jklvBPuKwasPyOFLZoioSrccTbaV77jdkfiN9VCXYm9h2dzha649KwMJPa5C1WzjmGyhW6dR8rbJilu6xZKi-wU8Qx6wi5LZgQ7nP8F4Qz1qiNBgYJ9KYxEmUjT5qiz5O8ZW015AOIa2MiNO5HXhnizxHdAiyUsHGwyD1wC9cXtm8wPbEbmcsMUQvqhX4wqSuWP-fDw9n2OJ_j3k7L3yZnIh-MLy63veXc21fOhPgM2JRzO3sAayueXdnrebzCxcLjJQfDBzZkPz5MAiFXb-RbCNwv7mYg2rBbnmGSWExoPzDmKtg2OI1oorv2tuNwt3b8jtLUpshvPCQXg"
}
2019-10-18 14:35:48,764:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/837052177 HTTP/1.1" 200 798
2019-10-18 14:35:48,765:DEBUG:acme.client:Received response:
HTTP 200
content-length: 798
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 69588500
date: Fri, 18 Oct 2019 19:35:48 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0002Y7rHx_YxYiZTLAz_Cn9SvIw3jqr7XlG-tl_ye4bgF78

{
  "identifier": {
    "type": "dns",
    "value": "genelab-galaxy.usra.edu"
  },
  "status": "pending",
  "expires": "2019-10-25T19:35:48Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/2amD_g",
      "token": "n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/zgjqFQ",
      "token": "n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/wxoYKg",
      "token": "n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak"
    }
  ]
}
2019-10-18 14:35:48,765:DEBUG:acme.client:Storing nonce: 0002Y7rHx_YxYiZTLAz_Cn9SvIw3jqr7XlG-tl_ye4bgF78
2019-10-18 14:35:48,766:INFO:certbot.auth_handler:Performing the following challenges:
2019-10-18 14:35:48,767:INFO:certbot.auth_handler:http-01 challenge for genelab-galaxy.usra.edu
2019-10-18 14:35:48,831:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: None in: /etc/httpd/conf/httpd.conf
2019-10-18 14:35:48,832:DEBUG:certbot_apache.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

2019-10-18 14:35:48,832:DEBUG:certbot_apache.http_01:writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>

2019-10-18 14:35:48,930:DEBUG:certbot.reverter:Creating backup of /etc/httpd/conf/httpd.conf
2019-10-18 14:35:52,331:INFO:certbot.auth_handler:Waiting for verification...
2019-10-18 14:35:52,333:DEBUG:acme.client:JWS payload:
{
  "type": "http-01",
  "resource": "challenge"
}
2019-10-18 14:35:52,337:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/2amD_g:
{
  "protected": "eyJub25jZSI6ICIwMDAyWTdySHhfWXhZaVpUTEF6X0NuOVN2SXczanFyN1hsRy10bF95ZTRiZ0Y3OCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwtdjMvODM3MDUyMTc3LzJhbURfZyIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82OTU4ODUwMCIsICJhbGciOiAiUlMyNTYifQ",
  "payload": "ewogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9",
  "signature": "vl0LLXgCp3F0Bq0rth8AOg-r_9nP1wDU2K3GDBjZjTZGm7Gs2MXB1ufknQWnDY6rjkDiqXKNpey9NpcgP0Nw-JXlqaVrssFGdRatp5zf85MzgiYSSFmLe9v-ewvQu3PKIwc2heSeXkxPvwHO8x4jm1cG1T_06eeFl5yMfvQ0zvt3k2LRZK8edz2q8VF2C-YOmvLkR25i8GeRgBs9ZDg47MxhH9rfL_f7x229MMBBCS1PTt6LJ38SDxSE4B-qwl5_NsQ0KaSwTy_Uc8ihFfVva7U2b50OjwMQ73vaU1epV28e1O7ddvicVFbc14IFRj-M3Yen2tDGqcF0uxcG2R7B2w"
}
2019-10-18 14:35:52,416:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/chall-v3/837052177/2amD_g HTTP/1.1" 200 184
2019-10-18 14:35:52,417:DEBUG:acme.client:Received response:
HTTP 200
content-length: 184
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/837052177>;rel="up"
location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/2amD_g
boulder-requester: 69588500
date: Fri, 18 Oct 2019 19:35:52 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0001RjWysEc1fbgiRTztQPNCOFxu6_6nvgCQrjE5fISRIjA

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/2amD_g",
  "token": "n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak"
}
2019-10-18 14:35:52,418:DEBUG:acme.client:Storing nonce: 0001RjWysEc1fbgiRTztQPNCOFxu6_6nvgCQrjE5fISRIjA
2019-10-18 14:35:53,420:DEBUG:acme.client:JWS payload:

2019-10-18 14:35:53,423:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/837052177:
{
  "protected": "eyJub25jZSI6ICIwMDAxUmpXeXNFYzFmYmdpUlR6dFFQTkNPRnh1Nl82bnZnQ1FyakU1ZklTUklqQSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvODM3MDUyMTc3IiwgImtpZCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzY5NTg4NTAwIiwgImFsZyI6ICJSUzI1NiJ9",
  "payload": "",
  "signature": "ziZtUdtKP4GAKQ2jDekdJPuyT6-yF-KvnZgGzjngLXG8oP9FSmFy8pvZBAOmQZuLvkccLeS_ATdjbQ-UHTDhM9VXZgFSQJ6mNCFImSFlnxD1i40NkNVJ3Z4guoz9YyHpdzAkrkmFkJed5hIdTU43g-JHtHXo1k1pvYfV2g_9NdjQVNclTNSa0ARGsCHrJLeDD0KMPi94kLb4sFP1fiz9jzLg5-8bIK11kjNqdHI-eqchNzDF-pQWBKsrl7-kMfReKR3KmtG9VCHD49ZFC3VtQE2PjJM3PET-OlNQck06maAOpWdPNG-wbLJ7gRTcqy8dv_BIIRxHsMYCpRCPhFwdCw"
}
2019-10-18 14:35:53,520:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/837052177 HTTP/1.1" 200 1678
2019-10-18 14:35:53,521:DEBUG:acme.client:Received response:
HTTP 200
content-length: 1678
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 69588500
date: Fri, 18 Oct 2019 19:35:53 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0002G1ujl68rFrP7J4kZgsViKBARGFdzw4WKsZ_ufQh63d8

{
  "identifier": {
    "type": "dns",
    "value": "genelab-galaxy.usra.edu"
  },
  "status": "invalid",
  "expires": "2019-10-25T19:35:48Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://genelab-galaxy.usra.edu/.well-known/acme-challenge/n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak [63.96.143.215]: \"\u003c!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\"\u003e\\n\u003chtml\u003e\u003chead\u003e\\n\u003ctitle\u003e403 Forbidden\u003c/title\u003e\\n\u003c/head\u003e\u003cbody\u003e\\n\u003ch1\u003eForbidden\u003c/h1\u003e\\n\u003cp\"",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/2amD_g",
      "token": "n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak",
      "validationRecord": [
        {
          "url": "http://genelab-galaxy.usra.edu/.well-known/acme-challenge/n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak",
          "hostname": "genelab-galaxy.usra.edu",
          "port": "80",
          "addressesResolved": [
            "63.96.143.215"
          ],
          "addressUsed": "63.96.143.215"
        }
      ]
    },
    {
      "type": "dns-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/zgjqFQ",
      "token": "n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak"
    },
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/837052177/wxoYKg",
      "token": "n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak"
    }
  ]
}
2019-10-18 14:35:53,521:DEBUG:acme.client:Storing nonce: 0002G1ujl68rFrP7J4kZgsViKBARGFdzw4WKsZ_ufQh63d8
2019-10-18 14:35:53,523:WARNING:certbot.auth_handler:Challenge failed for domain genelab-galaxy.usra.edu
2019-10-18 14:35:53,523:INFO:certbot.auth_handler:http-01 challenge for genelab-galaxy.usra.edu
2019-10-18 14:35:53,524:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: genelab-galaxy.usra.edu
Type:   unauthorized
Detail: Invalid response from http://genelab-galaxy.usra.edu/.well-known/acme-challenge/n_ZazQ_28TB4Xhf4k_s01ff4exKZm07r2_QAhibaEak [63.96.143.215]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2019-10-18 14:35:53,524:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 154, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

2019-10-18 14:35:53,525:DEBUG:certbot.error_handler:Calling registered functions
2019-10-18 14:35:53,525:INFO:certbot.auth_handler:Cleaning up challenges
2019-10-18 14:35:54,058:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.38.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1378, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1133, in run
    certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 121, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 405, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 348, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 384, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 154, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

I think you are blocking dotfiles, and Certbot’s Apache plugin can’t deal with it.

$ curl -i http://genelab-galaxy.usra.edu/.dotfolder/
HTTP/1.1 403 Forbidden

If we remove the leading dot:

$ curl -i http://genelab-galaxy.usra.edu/dotfolder/
HTTP/1.1 404 Not Found

Can you locate this rule and add an exclusion for /.well-known/*?

While we probably have found one, I am not aware of a situation where we can’t deal with a configuration that normal blocks dotfiles in our Apache plugin so knowing what is causing this would be helpful because it could allow us to fix the problem.

On Apache 2.4, the way we currently do this is enable mod_rewrite and add this to the top of all matching vhosts:

RewriteEngine on
RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

and this to the bottom of the vhost:

<Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
</Directory>
<Location /.well-known/acme-challenge>
            Require all granted
</Location>

The challenge file is located in /var/lib/letsencrypt/http_challenges.

If anyone is able to reproduce the problem, you can make Certbot stop after setting up the challenge so you can try to debug it with the --debug-challenges flag.

It might not be relevant in this case, but things like mod_security2 rules will trump what Certbot currently does, thanks to hooking as APR_HOOK_FIRST.

A basic repro is certbot --apache with e.g.:

SecRule REQUEST_URI "^/\..*" "id:1234,deny,log,msg:'block dotfile'"

It’s not something I’ve ever seen in any ruleset, but sysadmin creativity is unlimited.

Are you using mod_security, @chawlak11?

No, I am not using mod_security.

Here is my httpd.conf file:

# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so 'log/access_log'
# with ServerRoot set to '/www' will be interpreted by the
# server as '/www/log/access_log', where as '/log/access_log' will be
# interpreted as '/log/access_log'.

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# Do not add a slash at the end of the directory path.  If you point
# ServerRoot at a non-local disk, be sure to specify a local disk on the
# Mutex directive, if file-based mutexes are used.  If you wish to share the
# same ServerRoot for multiple httpd daemons, you will need to change at
# least PidFile.
#
ServerName www.genelab-galaxy.usra.edu

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. admin@your-domain.com
#
ServerAdmin root@localhost

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
ServerName www.genelab-galaxy.usra.edu:80



ServerRoot "/etc/httpd"

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses.
#
#Listen 12.34.56.78:80
Listen 80

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
Include conf.modules.d/*.conf
LoadModule rewrite_module modules/mod_rewrite.so
#LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
#LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule     proxy_module            modules/mod_proxy.so
LoadModule     proxy_http_module       modules/mod_proxy_http.so
LoadModule     proxy_uwsgi_module      modules/mod_proxy_uwsgi.so
#LoadModule     jk_module               modules/mod_jk.so
LoadModule      headers_module          modules/mod_headers.so
LoadModule     proxy_http_module       modules/mod_proxy_http.so
LoadModule     proxy_uwsgi_module      modules/mod_proxy_uwsgi.so
LoadModule      deflate_module          modules/mod_deflate.so
LoadModule      expires_module          modules/mod_expires.so
LoadModule     ssl_module                       modules/mod_ssl.so
#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User apache
Group apache

# 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition.  These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#
#SSLProtocol             all -SSLv3
#SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256
#SSLHonorCipherOrder     on
#SSLCompression          off
#SSLSessionTickets       off

# OCSP stapling
#SSLUseStapling          on
#SSLStaplingResponderTimeout 5
#SSLStaplingReturnResponderErrors off
#SSLStaplingCache        shmcb:/var/run/ocsp(128000)

#
# Deny access to the entirety of your server's filesystem. You must
# explicitly permit access to web content directories in other
# <Directory> blocks below.
#
<Directory />
        # By default don't use any options for performance/security.
        Options None
        # Leave AllowOverride to None for performance/security
        AllowOverride None
                #Order deny,allow
        #Deny from all
        Order allow,deny
        Allow from all
</Directory>

# Don't allow clients to access hidden files.  This includes access and password files
# like .htaccess and .htpasswd.
<Files ~ "^\.">
        Order allow,deny
        Deny from all
        Satisfy All
</Files>

# Main document root.
<Directory /data/docroot>
        Order allow,deny
        Allow from all
</Directory>

<Directory /data/galaxy/static>
    Order allow,deny
    Options FollowSymLinks
    Allow from all
</Directory>


Alias /static /configure-galaxy/static
#Alias /galaxy/static /data/galaxy/static

AllowEncodedSlashes NoDecode
RewriteRule ^/configure-galaxy/static/style/(.*) /data/galaxy/static/june_2007_style/blue/$1 [L]
RewriteRule ^/configure-galaxy/static/scripts/(.*) /data/galaxy/static/scripts/$1 [L]
RewriteRule ^/configure-galaxy/static/(.*) /data/galaxy/static/$1 [L]
RewriteRule ^/configure-galaxy/favicon.ico /data/galaxy/favicon.ico [L]
RewriteRule ^/configure-galaxy/images/(.*) /data/galaxy/static/images/$1 [L]
RewriteRule ^/configure-galaxy/robots.txt /data/galaxy/static/robots.txt [L]

#Configure Galaxy server for SSL access behind a httpd web proxy
<Location "/configure-galaxy/static">
ProxyPass !
RequestHeader set X-URL-SCHEME http
Options FollowSymLinks
Allow from all
Order allow,deny
# Allow browsers to cache everything from /static for 6 hours
ExpiresActive On
ExpiresDefault "access plus 6 hours"
</Location>

<Location "/configure-galaxy">
#ProxyPass unix:///data/galaxy/var/uwsgi.sock|uwsgi://
#ProxyPassReverse unix:///data/galaxy/var/uwsgi.sock|uwsgi://
ProxyPass http://127.0.0.1:8080
ProxypassReverse http://127.0.0.1:8080
RequestHeader set X-URL-SCHEME http
Options +FollowSymLinks -SymLinksIfOwnerMatch
Order Allow,Deny
Allow from all

# Compress all uncompressed content.
SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary
</Location>

Alias /favicon.ico /data/galaxy/static/favicon.ico
Alias /meatball-favicon.ico /data/galaxy/static/images/meatballfavicon.ico
Alias /icons /icons



<VirtualHost *:80>
    RewriteEngine On
    RewriteOptions Inherit
ServerAlias genelab-galaxy.usra.edu
ServerAlias wwww.genelab-galaxy.usra.edu


#these are all redirects that should be removed when configured for production all routes should route to /galaxy/*
Redirect permanent "/static" "/configure-galaxy/static"
Redirect permanent "/api" "/configure-galaxy/api"
Redirect permanent "/history" "/configure-galaxy/history"
Redirect permanent "/login" "/configure-galaxy/login"
Redirect permanent "/library" "/configure-galaxy/library"
Redirect permanent "/user" "/configure-galaxy/user"
Redirect permanent "/tool_runner" "/configure-galaxy/tool_runner"

#while galaxy is being configured route calls to /galaxy to the base server
Redirect permanent "/galaxy" "/"

# Enable HSTS
    Header always set Strict-Transport-Security "max-age=15552000; includeSubdomains"

    # use a variable for convenience
    Define galaxy_root /data/galaxy
   # enable compression on all relevant types
    AddOutputFilterByType DEFLATE text/html text/plain text/xml
    AddOutputFilterByType DEFLATE text/css
    AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
    AddOutputFilterByType DEFLATE application/rss+xml
    AddOutputFilterByType DEFLATE application/xml
    AddOutputFilterByType DEFLATE application/json

#    ProxyPass "/configure-galaxy" unix:///data/galaxy/var/uwsgi.sock|uwsgi://
 #   ProxypassReverse "/configure-galaxy" unix:///data/galaxy/var/uwsgi.sock|uwsgi://
#ProxyPass "/configure-galaxy" http://127.0.0.1:8080/
#ProxypassReverse "/configure-galaxy" http://127.0.0.1:8080/
    # Galaxy needs to know that this is https for generating URLs
    RequestHeader set X-URL-SCHEME "%{REQUEST_SCHEME}e"

    # allow up to 3 minutes for Galaxy to respond to slow requests before timing out
    ProxyTimeout 180

    # proxy all requests not matching other locations to uWSGI
    #ProxyPass /configure-galaxy unix:///usr/sbin/uwsgi
    # or uWSGI on a TCP socket
    #ProxyPass / uwsgi://127.0.0.1:8080



    # serve visualization and interactive environment plugin static content
    <Directory "/data/galaxy/config/plugins/(.+)/(.+)/static">
        AllowOverride None
        Require all granted
    </Directory>
    RewriteRule ^/plugins/(.+)/(.+)/static/(.*)$ /data/galaxy/config/plugins/$1/$2/static/$3 [L]
</VirtualHost>



#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/galaxy/html"

#
# Relax access to content within /var/www.
#
<Directory "/var/www/galaxy/html">
    AllowOverride none
    # Allow open access:
    Require all granted
</Directory>
# Further relax access to the default document root:
<Directory "/var/www/html">
    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.4/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks
    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride none

    #
    # Controls who can get stuff from this server.
    #
    Require all granted
</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<Files ".ht*">
    Require all denied
</Files>
#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog "logs/error_log"

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    #CustomLog "logs/access_log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    CustomLog "logs/access_log" combined
</IfModule>
<IfModule alias_module>
    #
    # Redirect: Allows you to tell clients about documents that used to
    # exist in your server's namespace, but do not anymore. The client
    # will make a new request for the document at its new location.
    # Example:
    # Redirect permanent /foo http://www.example.com/bar

    #
    # Alias: Maps web paths into filesystem paths and is used to
    # access content that does not live under the DocumentRoot.
    # Example:
    # Alias /webpath /full/filesystem/path
    #
    # If you include a trailing / on /webpath then the server will
    # require it to be present in the URL.  You will also likely
    # need to provide a <Directory> section to allow access to
    # the filesystem path.

    #
    # ScriptAlias: This controls which directories contain server scripts.
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the target directory are treated as applications and
    # run by the server when requested rather than as documents sent to the
    # client.  The same rules about trailing "/" apply to ScriptAlias
    # directives as to Alias.
    #
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
    AllowOverride All
    Options None
    Require all granted
</Directory>
<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig /etc/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    # For type maps (negotiated resources):
    #AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>
#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default.  To use the
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
AddDefaultCharset UTF-8

<IfModule mime_magic_module>
    #
    # The mod_mime_magic module allows the server to use various hints from the
    # contents of the file itself to determine its type.  The MIMEMagicFile
    # directive tells the module where the hint definitions are located.
    #
    MIMEMagicFile conf/magic
</IfModule>

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# EnableMMAP and EnableSendfile: On systems that support it,
# memory-mapping or the sendfile syscall may be used to deliver
# files.  This usually improves server performance, but must
# be turned off when serving from networked-mounted
# filesystems or if support for these functions is otherwise
# broken on your system.
# Defaults if commented: EnableMMAP On, EnableSendfile Off
#
#EnableMMAP off
EnableSendfile on

# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.


#when 443 ssl si configed here comment this out
IncludeOptional conf.d/*.conf
#IncludeOptional sites-enabled/*.conf

I have added this:

RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

and this to the bottom of the vhost:

and also

<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

Still no luck !!

@chawlak11, @bmw was describing what Certbot itself does (temporarily) to your Apache configuration, and not something that you should have to do to your configuration in order to use Certbot.

Could you post your existing Apache configuration, including other files that may be part of it?

In case it saves somebody trying, I tried using the Apache configuration from a few posts up, and Certbot (0.31.0) issued the certificate successfully.

That could suggest that the problem may be dependent on other files, such as those in conf.d/.

I’d be curious to see whether commenting this out helps though:

# Don't allow clients to access hidden files.  This includes access and password files
# like .htaccess and .htpasswd.
<Files ~ "^\.">
        Order allow,deny
        Deny from all
        Satisfy All
</Files>

Edit: tried with 0.39.0 too.

Looks like dotfiles isn’t it after all, based on the post below.

Just tried commenting it but no luck.

Here is my httpd.conf file

sudo cat /etc/httpd/conf/httpd.conf

This is the main Apache HTTP server configuration file. It contains the

configuration directives that give the server its instructions.

See URL:http://httpd.apache.org/docs/2.4/ for detailed information.

In particular, see

URL:http://httpd.apache.org/docs/2.4/mod/directives.html

for a discussion of each configuration directive.

Do NOT simply read the instructions in here without understanding

what they do. They’re here only as hints or reminders. If you are unsure

consult the online docs. You have been warned.

Configuration and logfile names: If the filenames you specify for many

of the server’s control files begin with “/” (or “drive:/” for Win32), the

server will use that explicit path. If the filenames do not begin

with “/”, the value of ServerRoot is prepended – so ‘log/access_log’

with ServerRoot set to ‘/www’ will be interpreted by the

server as ‘/www/log/access_log’, where as ‘/log/access_log’ will be

interpreted as ‘/log/access_log’.

ServerRoot: The top of the directory tree under which the server’s

configuration, error, and log files are kept.

Do not add a slash at the end of the directory path. If you point

ServerRoot at a non-local disk, be sure to specify a local disk on the

Mutex directive, if file-based mutexes are used. If you wish to share the

same ServerRoot for multiple httpd daemons, you will need to change at

least PidFile.

ServerName www.genelab-galaxy.usra.edu

ServerAdmin: Your address, where problems with the server should be

e-mailed. This address appears on some server-generated pages, such

as error documents. e.g. admin@your-domain.com

ServerAdmin root@localhost

ServerName gives the name and port that the server uses to identify itself.

This can often be determined automatically, but we recommend you specify

it explicitly to prevent problems during startup.

If your host doesn’t have a registered DNS name, enter its IP address here.

ServerName www.genelab-galaxy.usra.edu:80

ServerRoot “/etc/httpd”

Listen: Allows you to bind Apache to specific IP addresses and/or

ports, instead of the default. See also the

directive.

Change this to Listen on specific IP addresses as shown below to

prevent Apache from glomming onto all bound IP addresses.

#Listen 12.34.56.78:80
Listen 80

Dynamic Shared Object (DSO) Support

To be able to use the functionality of a module which was built as a DSO you

have to place corresponding `LoadModule’ lines at this location so the

directives contained in it are actually available before they are used.

Statically compiled modules (those listed by `httpd -l’) do not need

to be loaded here.

Example:

LoadModule foo_module modules/mod_foo.so

Include conf.modules.d/*.conf
LoadModule rewrite_module modules/mod_rewrite.so
#LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
#LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
#LoadModule jk_module modules/mod_jk.so
LoadModule headers_module modules/mod_headers.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule expires_module modules/mod_expires.so
LoadModule ssl_module modules/mod_ssl.so

If you wish httpd to run as a different user or group, you must run

httpd as root initially and it will switch.

User/Group: The name (or #number) of the user/group to run httpd as.

It is usually good practice to create a dedicated user and group for

running httpd, as with most system services.

User apache
Group apache

‘Main’ server configuration

The directives in this section set up the values used by the ‘main’

server, which responds to any requests that aren’t handled by a

definition. These values also provide defaults for

any containers you may define later in the file.

All of these directives may appear inside containers,

in which case these default settings will be overridden for the

virtual host being defined.

#SSLProtocol all -SSLv3
#SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256
#SSLHonorCipherOrder on
#SSLCompression off
#SSLSessionTickets off

OCSP stapling

#SSLUseStapling on
#SSLStaplingResponderTimeout 5
#SSLStaplingReturnResponderErrors off
#SSLStaplingCache shmcb:/var/run/ocsp(128000)

Deny access to the entirety of your server’s filesystem. You must

explicitly permit access to web content directories in other

blocks below.

# By default don't use any options for performance/security. Options None # Leave AllowOverride to None for performance/security AllowOverride None #Order deny,allow #Deny from all Order allow,deny Allow from all

Don’t allow clients to access hidden files. This includes access and password files

like .htaccess and .htpasswd.

<Files ~ “^.”>
Order allow,deny
Deny from all
Satisfy All

Main document root.

<Directory /data/docroot>
Order allow,deny
Allow from all

<Directory /data/galaxy/static>
Order allow,deny
Options FollowSymLinks
Allow from all

Alias /static /configure-galaxy/static
#Alias /galaxy/static /data/galaxy/static

AllowEncodedSlashes NoDecode
RewriteRule ^/configure-galaxy/static/style/(.) /data/galaxy/static/june_2007_style/blue/$1 [L]
RewriteRule ^/configure-galaxy/static/scripts/(.
) /data/galaxy/static/scripts/$1 [L]
RewriteRule ^/configure-galaxy/static/(.) /data/galaxy/static/$1 [L]
RewriteRule ^/configure-galaxy/favicon.ico /data/galaxy/favicon.ico [L]
RewriteRule ^/configure-galaxy/images/(.
) /data/galaxy/static/images/$1 [L]
RewriteRule ^/configure-galaxy/robots.txt /data/galaxy/static/robots.txt [L]

#Configure Galaxy server for SSL access behind a httpd web proxy
<Location “/configure-galaxy/static”>
ProxyPass !
RequestHeader set X-URL-SCHEME http
Options FollowSymLinks
Allow from all
Order allow,deny

Allow browsers to cache everything from /static for 6 hours

ExpiresActive On
ExpiresDefault “access plus 6 hours”

<Location “/configure-galaxy”>
#ProxyPass unix:///data/galaxy/var/uwsgi.sock|uwsgi://
#ProxyPassReverse unix:///data/galaxy/var/uwsgi.sock|uwsgi://
ProxyPass http://127.0.0.1:8080
ProxypassReverse http://127.0.0.1:8080
RequestHeader set X-URL-SCHEME http
Options +FollowSymLinks -SymLinksIfOwnerMatch
Order Allow,Deny
Allow from all

Compress all uncompressed content.

SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png) no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2) no-gzip dont-vary
SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary

Alias /favicon.ico /data/galaxy/static/favicon.ico
Alias /meatball-favicon.ico /data/galaxy/static/images/meatballfavicon.ico
Alias /icons /icons

<VirtualHost *:80>
RewriteEngine On
RewriteOptions Inherit
ServerAlias genelab-galaxy.usra.edu
ServerAlias wwww.genelab-galaxy.usra.edu

#these are all redirects that should be removed when configured for production all routes should route to /galaxy/*
Redirect permanent “/static” “/configure-galaxy/static”
Redirect permanent “/api” “/configure-galaxy/api”
Redirect permanent “/history” “/configure-galaxy/history”
Redirect permanent “/login” “/configure-galaxy/login”
Redirect permanent “/library” “/configure-galaxy/library”
Redirect permanent “/user” “/configure-galaxy/user”
Redirect permanent “/tool_runner” “/configure-galaxy/tool_runner”

#while galaxy is being configured route calls to /galaxy to the base server
Redirect permanent “/galaxy” “/”

Enable HSTS

Header always set Strict-Transport-Security "max-age=15552000; includeSubdomains"

# use a variable for convenience
Define galaxy_root /data/galaxy

enable compression on all relevant types

AddOutputFilterByType DEFLATE text/html text/plain text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/json

ProxyPass “/configure-galaxy” unix:///data/galaxy/var/uwsgi.sock|uwsgi://

ProxypassReverse “/configure-galaxy” unix:///data/galaxy/var/uwsgi.sock|uwsgi://

#ProxyPass “/configure-galaxy” http://127.0.0.1:8080/
#ProxypassReverse “/configure-galaxy” http://127.0.0.1:8080/
# Galaxy needs to know that this is https for generating URLs
RequestHeader set X-URL-SCHEME “%{REQUEST_SCHEME}e”

# allow up to 3 minutes for Galaxy to respond to slow requests before timing out
ProxyTimeout 180

# proxy all requests not matching other locations to uWSGI
#ProxyPass /configure-galaxy unix:///usr/sbin/uwsgi
# or uWSGI on a TCP socket
#ProxyPass / uwsgi://127.0.0.1:8080



# serve visualization and interactive environment plugin static content
<Directory "/data/galaxy/config/plugins/(.+)/(.+)/static">
    AllowOverride None
    Require all granted
</Directory>

RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
RewriteRule ^/plugins/(.+)/(.+)/static/(.*)$ /data/galaxy/config/plugins/$1/$2/static/$3 [L]
<Directory /var/lib/letsencrypt/http_challenges>
        Require all granted
</Directory>
<Location /.well-known/acme-challenge>
        Require all granted
</Location>

Note that from this point forward you must specifically allow

particular features to be enabled - so if something’s not working as

you might expect, make sure that you have specifically enabled it

below.

DocumentRoot: The directory out of which you will serve your

documents. By default, all requests are taken from this directory, but

symbolic links and aliases may be used to point to other locations.

DocumentRoot “/var/www/galaxy/html”

Relax access to content within /var/www.

<Directory “/var/www/galaxy/html”>
AllowOverride none
# Allow open access:
Require all granted

Further relax access to the default document root:

<Directory “/var/www/html”>
#
# Possible values for the Options directive are “None”, “All”,
# or any combination of:
# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that “MultiViews” must be named explicitly — “Options All”
# doesn’t give it to you.
#
# The Options directive is both complicated and important. Please see
# http://httpd.apache.org/docs/2.4/mod/core.html#options
# for more information.
#
Options Indexes FollowSymLinks
#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be “All”, “None”, or any combination of the keywords:
# Options FileInfo AuthConfig Limit
#
AllowOverride none

#
# Controls who can get stuff from this server.
#
Require all granted

DirectoryIndex: sets the file that Apache will serve if a directory

is requested.

DirectoryIndex index.html

The following lines prevent .htaccess and .htpasswd files from being

viewed by Web clients.

<Files “.ht*”>
Require all denied

ErrorLog: The location of the error log file.

If you do not specify an ErrorLog directive within a

container, error messages relating to that virtual host will be

logged here. If you do define an error logfile for a

container, that host’s errors will be logged there and not here.

ErrorLog “logs/error_log”

LogLevel: Control the number of messages logged to the error_log.

Possible values include: debug, info, notice, warn, error, crit,

alert, emerg.

LogLevel warn

# # The following directives define some format nicknames for use with # a CustomLog directive (see below). # LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common
<IfModule logio_module>
  # You need to enable mod_logio.c to use %I and %O
  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>

#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here.  Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
#CustomLog "logs/access_log" common

#
# If you prefer a logfile with access, agent, and referer information
# (Combined Logfile Format) you can use the following directive.
#
CustomLog "logs/access_log" combined
# # Redirect: Allows you to tell clients about documents that used to # exist in your server's namespace, but do not anymore. The client # will make a new request for the document at its new location. # Example: # Redirect permanent /foo http://www.example.com/bar
#
# Alias: Maps web paths into filesystem paths and is used to
# access content that does not live under the DocumentRoot.
# Example:
# Alias /webpath /full/filesystem/path
#
# If you include a trailing / on /webpath then the server will
# require it to be present in the URL.  You will also likely
# need to provide a <Directory> section to allow access to
# the filesystem path.

#
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the target directory are treated as applications and
# run by the server when requested rather than as documents sent to the
# client.  The same rules about trailing "/" apply to ScriptAlias
# directives as to Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

“/var/www/cgi-bin” should be changed to whatever your ScriptAliased

CGI directory exists, if you have that configured.

<Directory “/var/www/cgi-bin”>
AllowOverride All
Options None
Require all granted


#
# TypesConfig points to the file containing the list of mappings from
# filename extension to MIME-type.
#
TypesConfig /etc/mime.types

#
# AddType allows you to add to or override the MIME configuration
# file specified in TypesConfig for specific file types.
#
#AddType application/x-gzip .tgz
#
# AddEncoding allows you to have certain browsers uncompress
# information on the fly. Note: Not all browsers support this.
#
#AddEncoding x-compress .Z
#AddEncoding x-gzip .gz .tgz
#
# If the AddEncoding directives above are commented-out, then you
# probably should define those extensions to indicate media types:
#
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz

#
# AddHandler allows you to map certain file extensions to "handlers":
# actions unrelated to filetype. These can be either built into the server
# or added with the Action directive (see below)
#
# To use CGI scripts outside of ScriptAliased directories:
# (You will also need to add "ExecCGI" to the "Options" directive.)
#
#AddHandler cgi-script .cgi

# For type maps (negotiated resources):
#AddHandler type-map var

#
# Filters allow you to process content before it is sent to the client.
#
# To parse .shtml files for server-side includes (SSI):
# (You will also need to add "Includes" to the "Options" directive.)
#
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
# # Specify a default charset for all content served; this enables # interpretation of all content as UTF-8 by default. To use the # default browser choice (ISO-8859-1), or to allow the META tags # in HTML content to override this choice, comment out this # directive: # AddDefaultCharset UTF-8 # # The mod_mime_magic module allows the server to use various hints from the # contents of the file itself to determine its type. The MIMEMagicFile # directive tells the module where the hint definitions are located. # MIMEMagicFile conf/magic

Customizable error responses come in three flavors:

1) plain text 2) local redirects 3) external redirects

Some examples:

#ErrorDocument 500 “The server made a boo boo.”
#ErrorDocument 404 /missing.html
#ErrorDocument 404 “/cgi-bin/missing_handler.pl”
#ErrorDocument 402 http://www.example.com/subscription_info.html

EnableMMAP and EnableSendfile: On systems that support it,

memory-mapping or the sendfile syscall may be used to deliver

files. This usually improves server performance, but must

be turned off when serving from networked-mounted

filesystems or if support for these functions is otherwise

broken on your system.

Defaults if commented: EnableMMAP On, EnableSendfile Off

#EnableMMAP off
EnableSendfile on

Supplemental configuration

Load config files in the “/etc/httpd/conf.d” directory, if any.

#when 443 ssl si configed here comment this out
IncludeOptional conf.d/.conf
#IncludeOptional sites-enabled/
.conf

[kchawla@galaxy ~]$