How To Force Certbot To Use Domain Name, Not Nameserver?


#1

I am using certbot-apache (Fedora 27) to implement Let’s Encrypt and cannot get certbot --apache or certbot --apache certonly to generate a certificate for my domain, - it insists on generating the certificates for ns2.insurgent.info, not insurgent.info; so, naturally enough, https subsequently fails for my domain because the website is insurgent.info, not ns2.insurgent.info.

Obviously I realise that I can change details in the various configuration files, but not in the encrypted certificate files, which makes fixing the issue manually out of the question.

The DNS of my site, and Apache configuration, are also absolutely fine, so the problem is not being caused by lack of propagation or bad DNS. Can anyone help with this, please?


#2

Can you provide more information?

Does Apache have a virtual host with the ServerName or ServerAlias set to ns2.insurgent.info?

To insurgent.info?


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

I ran this command:

It produced this output:

My web server is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):


#3

Thanks for the reply, - as per your advice I have added ServerAlias to my VirtualHost blocks, and I use the default gateway IP address (IPv4) for the ServerName. The server is a home server, running BIND with Apache 2.4, and I have full shell access. Everything is command line, with no control panel.

The Apache VirtualHost blocks are as follows:

<VirtualHost 46.102.204.227:80>
  ServerAdmin info@some.info
  DocumentRoot "/var/www"
  ServerAlias ns2.insurgent.info
</VirtualHost>

<VirtualHost [2a00:b900:10a4:1::4]:80>
  ServerAdmin info@some.info
  DocumentRoot "/var/www"
  ServerAlias ns2.insurgent.info
</VirtualHost>



The latest result of certbot --apache is as follows:

IMPORTANT NOTES:

 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/ns2.insurgent.info/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/ns2.insurgent.info/privkey.pem
   Your cert will expire on 2018-12-31. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

I had to add in ns2.insurgent.info as ServerName in each of the VirtualHost blocks, with no ServerAlias. The problem is that https is still causing an untrusted certificate error in Firefox:

insurgent.info uses an invalid security certificate. The certificate is only valid for ns2.insurgent.info Error code: SSL_ERROR_BAD_CERT_DOMAIN

…so I am, effectively, back at the beginning and, again, needing some way of forcing certbot to accept that my domain is insurgent.info and not ns2.insurgent.info.


#4

Set insurgent.info in ServerName or ServerAlias, or pass “-d insurgent.info” to Certbot along with the other names you want to use.


#5

Thanks, but neither of those solutions worked (tested with insurgent.info as both ServerName and ServerAlias in httpd.conf):

[root@localhost ~]# certbot -d insurgent.info --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for insurgent.info
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. insurgent.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://insurgent.info/.well-known/acme-challenge/25xHaHJZFoSf7ymoCXxi7oYEC0Z0YaX3x943IA03fBE: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: insurgent.info
   Type:   unauthorized
   Detail: Invalid response from
   http://insurgent.info/.well-known/acme-challenge/25xHaHJZFoSf7ymoCXxi7oYEC0Z0YaX3x943IA03fBE:
   "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

#6

Can you see if the ns2 name is mentioned anywhere in your Apache configuration?


#7

No, it is not. - I only ever add specific details like that when debugging or securing things specifically. - I can truthfully say that my DNS is absolutely 100% correct (I have spent a lot of time and effort on getting it right), so it looks very much as though Certbot is unable to parse DNS records properly and requires kluged configurations to make it work.


#8

So I find this extremely peculiar because you’re the only person I’ve ever found who has had certbot --apache result in a certificate for the name server. The domain names that the certificate covers are obtained by parsing the Apache configuration rather than querying DNS at all!

This makes it seem that you’ve found an extremely unusual bug in Certbot that most people don’t manage to trigger.

Could you please post the log files from /var/log/letsencrypt and the whole Apache configuration from /etc/apache2? (maybe it’s /etc/httpd on Fedora?)


#9

Fair enough, - here’s the log file:

2018-10-02 04:10:41,307:DEBUG:certbot.main:certbot version: 0.27.1
2018-10-02 04:10:41,308:DEBUG:certbot.main:Arguments: ['--apache']
2018-10-02 04:10:41,309:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-10-02 04:10:41,323:DEBUG:certbot.log:Root logging level set at 20
2018-10-02 04:10:41,324:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-10-02 04:10:41,324:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2018-10-02 04:10:41,422:DEBUG:certbot_apache.configurator:Apache version is 2.4.34
2018-10-02 04:10:41,726:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7f1517432278>
Prep: True
2018-10-02 04:10:41,727:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_centos.CentOSConfigurator object at 0x7f1517432278> and installer <certbot_apache.override_centos.CentOSConfigurator object at 0x7f1517432278>
2018-10-02 04:10:41,728:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2018-10-02 04:10:41,755:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/43116366', new_authzr_uri=None, terms_of_service=None), d64b89bcac55f990f8e5f69ed75ac4d7, Meta(creation_dt=datetime.datetime(2018, 10, 2, 2, 22, 13, tzinfo=<UTC>), creation_host='localhost.localdomain'))>
2018-10-02 04:10:41,757:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2018-10-02 04:10:41,759:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2018-10-02 04:10:42,221:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2018-10-02 04:10:42,223:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 658
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 02 Oct 2018 03:10:42 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 02 Oct 2018 03:10:42 GMT
Connection: keep-alive

{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
  "wY8FD66Kjsw": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
}
2018-10-02 04:10:47,046:INFO:certbot.main:Obtaining a new certificate
2018-10-02 04:10:47,125:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0005_key-certbot.pem
2018-10-02 04:10:47,128:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0005_csr-certbot.pem
2018-10-02 04:10:47,129:DEBUG:acme.client:Requesting fresh nonce
2018-10-02 04:10:47,129:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-order.
2018-10-02 04:10:47,327:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-order HTTP/1.1" 405 0
2018-10-02 04:10:47,328:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 103
Allow: POST
Replay-Nonce: xLplPkilzr9SSwmAm4gbXRUbeBUFICFdAAO6gmcA3pM
Expires: Tue, 02 Oct 2018 03:10:47 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 02 Oct 2018 03:10:47 GMT
Connection: keep-alive


2018-10-02 04:10:47,329:DEBUG:acme.client:Storing nonce: xLplPkilzr9SSwmAm4gbXRUbeBUFICFdAAO6gmcA3pM
2018-10-02 04:10:47,329:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "insurgent.info"\n    }\n  ],\n  "status": "pending",\n  "resource": "new-order"\n}'
2018-10-02 04:10:47,336:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNDMxMTYzNjYiLCAibm9uY2UiOiAieExwbFBraWx6cjlTU3dtQW00Z2JYUlViZUJVRklDRmRBQU82Z21jQTNwTSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIn0",
  "signature": "T_SgJhuJ9PRFIzlKWO7nWX_R9c3hN5TE7logAiskj2iTlFwFA4SnG0naQNY0I75eZI0MGnUwHuWbozfmRgd9_EfCXR4GOtC2NoyBoreMZy1LdnoN7VN786oG-1Fht-n0-rBRRuSoWH4-d1Y1rTczn0VovAMT8mCWViNd7AYi9AyOxSbG3bWXPGTaP1dB8PAtyHqcBTBZbPz1-gFzVKuHS4XKFqgT-B8qdi51SnnMbIG8K4iP7mE1c0lKBYdTMwfrUbsAza1nC7fM4LLiy83BewiL2eX2uGa-MpII1laFKFGPzIDsPMgNYkvGcYMN9GMlt3jvbla9u_sYTqTinCKzsQ",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImluc3VyZ2VudC5pbmZvIgogICAgfQogIF0sCiAgInN0YXR1cyI6ICJwZW5kaW5nIiwKICAicmVzb3VyY2UiOiAibmV3LW9yZGVyIgp9"
}
2018-10-02 04:10:47,588:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 372
2018-10-02 04:10:47,589:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 372
Boulder-Requester: 43116366
Location: https://acme-v02.api.letsencrypt.org/acme/order/43116366/93768647
Replay-Nonce: T1_xurpNGUjVGZn9Z-NiLAsZdzxsu1CAl5S98JuERnU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 02 Oct 2018 03:10:47 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 02 Oct 2018 03:10:47 GMT
Connection: keep-alive

{
  "status": "pending",
  "expires": "2018-10-09T03:10:47.458612831Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "insurgent.info"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/43116366/93768647"
}
2018-10-02 04:10:47,589:DEBUG:acme.client:Storing nonce: T1_xurpNGUjVGZn9Z-NiLAsZdzxsu1CAl5S98JuERnU
2018-10-02 04:10:47,590:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/acme/authz/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY.
2018-10-02 04:10:47,798:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /acme/authz/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY HTTP/1.1" 200 906
2018-10-02 04:10:47,799:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 906
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 02 Oct 2018 03:10:47 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 02 Oct 2018 03:10:47 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "insurgent.info"
  },
  "status": "pending",
  "expires": "2018-10-09T03:10:47Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815197",
      "token": "q8QVCybkKetN0AxylxvKkFIop2oZL2vD-wV5LvUKb-g"
    },
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815198",
      "token": "Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815199",
      "token": "eU5H7gpM2vPqlHUhBeJgpAtYPkjMSBiWOnlnCvXUo98"
    }
  ]
}
2018-10-02 04:10:47,801:INFO:certbot.auth_handler:Performing the following challenges:
2018-10-02 04:10:47,801:INFO:certbot.auth_handler:http-01 challenge for insurgent.info
2018-10-02 04:10:47,833:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: insurgent.info in: /etc/httpd/conf/httpd.conf
2018-10-02 04:10:47,834:DEBUG:certbot_apache.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
    
2018-10-02 04:10:47,834:DEBUG:certbot_apache.http_01:writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>
    
2018-10-02 04:10:48,196:DEBUG:certbot.reverter:Creating backup of /etc/httpd/conf/httpd.conf
2018-10-02 04:10:51,436:INFO:certbot.auth_handler:Waiting for verification...
2018-10-02 04:10:51,438:DEBUG:acme.client:JWS payload:
b'{\n  "resource": "challenge",\n  "keyAuthorization": "Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g.nioTlIhxgyQmLQJ6fyZeM84dZdQlpVs3XslrkliewQ0",\n  "type": "http-01"\n}'
2018-10-02 04:10:51,443:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815198:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNDMxMTYzNjYiLCAibm9uY2UiOiAiVDFfeHVycE5HVWpWR1puOVotTmlMQXNaZHp4c3UxQ0FsNVM5OEp1RVJuVSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGxlbmdlL0U1WUFIOG9DOWhXNUNqQWZlZmJGZktrdDNIRmNGY0cxOE1aWHVlQVNVcVkvNzg2OTgxNTE5OCJ9",
  "signature": "qGyPAZ9tjGnxxttoRGH34g5ZMkIkspdqm4TMe72nwKGC5ppnGO4jtgXOVOB2rYEmzzCnd58u_8VM-CobKKxkFsoTD1vW9WdJ6rJWBmfMc7XqfsoSHyUvlfycZAidj8Lpn1uU0H54Vjlo4uHaBnCEXBjTjV5Af6s90E32AutjUQl7kw74FfuM8KwGGJOr_EOFpmL9dyKJPQ_gLOiXbPRVin0t1SOcAJxvxtYLIxIPMMuEvAMRrl70ACYDyCQq1RvYGiWd5Ql9WFu1IzQ_oJ8GcoIO4RKEbJ4rOiTQvzqzePLEkfHp5RzfeAzuye_RP-_Wqxdugxio7fH4eFgWsqJXjA",
  "payload": "ewogICJyZXNvdXJjZSI6ICJjaGFsbGVuZ2UiLAogICJrZXlBdXRob3JpemF0aW9uIjogIk1mXzJQR3NiNVNTQTRvdDJGcEJ0dlU1ZEtQdndObUlZLXJWNm54aXVBMmcubmlvVGxJaHhneVFtTFFKNmZ5WmVNODRkWmRRbHBWczNYc2xya2xpZXdRMCIsCiAgInR5cGUiOiAiaHR0cC0wMSIKfQ"
}
2018-10-02 04:10:51,672:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815198 HTTP/1.1" 200 223
2018-10-02 04:10:51,674:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 223
Boulder-Requester: 43116366
Link: <https://acme-v02.api.letsencrypt.org/acme/authz/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY>;rel="up"
Location: https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815198
Replay-Nonce: x_nd1MHDh-OH_nSLjoPGElojNSdurw0aK9o-UBDKmyQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 02 Oct 2018 03:10:51 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 02 Oct 2018 03:10:51 GMT
Connection: keep-alive

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815198",
  "token": "Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g"
}
2018-10-02 04:10:51,674:DEBUG:acme.client:Storing nonce: x_nd1MHDh-OH_nSLjoPGElojNSdurw0aK9o-UBDKmyQ
2018-10-02 04:10:54,679:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/acme/authz/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY.
2018-10-02 04:10:54,893:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /acme/authz/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY HTTP/1.1" 200 1785
2018-10-02 04:10:54,894:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 1785
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 02 Oct 2018 03:10:54 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 02 Oct 2018 03:10:54 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "insurgent.info"
  },
  "status": "invalid",
  "expires": "2018-10-09T03:10:47Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815197",
      "token": "q8QVCybkKetN0AxylxvKkFIop2oZL2vD-wV5LvUKb-g"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://insurgent.info/.well-known/acme-challenge/Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g: \"\u003c!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\"\u003e\\n\u003chtml\u003e\u003chead\u003e\\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\\n\u003c/head\u003e\u003cbody\u003e\\n\u003ch1\u003eNot Found\u003c/h1\u003e\\n\u003cp\"",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815198",
      "token": "Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g",
      "validationRecord": [
        {
          "url": "http://insurgent.info/.well-known/acme-challenge/Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g",
          "hostname": "insurgent.info",
          "port": "80",
          "addressesResolved": [
            "46.102.204.227",
            "2a00:b900:10a4:1::4"
          ],
          "addressUsed": "2a00:b900:10a4:1::4"
        }
      ]
    },
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/E5YAH8oC9hW5CjAfefbFfKkt3HFcFcG18MZXueASUqY/7869815199",
      "token": "eU5H7gpM2vPqlHUhBeJgpAtYPkjMSBiWOnlnCvXUo98"
    }
  ]
}
2018-10-02 04:10:54,897:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: insurgent.info
Type:   unauthorized
Detail: Invalid response from http://insurgent.info/.well-known/acme-challenge/Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2018-10-02 04:10:54,898:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 155, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 226, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. insurgent.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://insurgent.info/.well-known/acme-challenge/Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

2018-10-02 04:10:54,898:DEBUG:certbot.error_handler:Calling registered functions
2018-10-02 04:10:54,898:INFO:certbot.auth_handler:Cleaning up challenges
2018-10-02 04:10:55,145:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.27.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 1124, in run
    certname, lineage)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 120, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3.6/site-packages/certbot/client.py", line 391, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python3.6/site-packages/certbot/client.py", line 334, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3.6/site-packages/certbot/client.py", line 370, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 155, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 226, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. insurgent.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://insurgent.info/.well-known/acme-challenge/Mf_2PGsb5SSA4ot2FpBtvU5dKPvwNmIY-rV6nxiuA2g: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

…and httpd.conf:

ServerRoot "/etc/httpd"
Listen 80

Include conf.modules.d/*.conf

User apache
Group apache

ServerAdmin root@localhost
ServerName 46.102.204.226:80

<VirtualHost 46.102.204.227:80>
  ServerAdmin info@some.info
  DocumentRoot "/var/insurgent"
  ServerName insurgent.info
</VirtualHost>

<VirtualHost [2a00:b900:10a4:1::4]:80>
  ServerAdmin info@some.info
  DocumentRoot "/var/insurgent"
  ServerName insurgent.info
</VirtualHost>

#
# Deny access to the entirety of your server's filesystem. You must
# explicitly permit access to web content directories in other 
# <Directory> blocks below.
#
<Directory />
    AllowOverride none
    Require all denied
</Directory>

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/insurgent"

#
# Relax access to content within /var/www.
#
<Directory "/var/insurgent">
    AllowOverride None
    # Allow open access:
    Require all granted

    Options +Includes +IncludesNoExec +Indexes +SymLinksIfOwnerMatch
	XBitHack Full

    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride None

    #
    # Controls who can get stuff from this server.
    #
    Require all granted
	
	RewriteEngine On
	RewriteCond %{THE_REQUEST} !HTTP/1.1$
	RewriteRule .* - [F]
	
	RewriteCond %{HTTP_REFERER} !^$
	RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?insurgent\.info/.*$ [NC]
	RewriteRule .*\.(jpg|jpeg|png|gif|odt)$ - [F,NC,L]

    # HTTP 1.1 protocol supports many request methods, including: GET, HEAD, POST,
    # PUT, DELETE, CONNECT, OPTIONS, TRACE, and PATCH. Web application will
    # usually only require GET, HEAD, and POST request methods. 
    #
    <LimitExcept GET POST HEAD>
      deny from all
    </LimitExcept>
</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.htm
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ".ht*">
    Require all denied
</Files>

ErrorLog "logs/error_log"
LogLevel warn

# <IfModule security2_module>
#    Include security/owasp-modsecurity-crs-3.0.2/crs-setup.conf
# </IfModule>

<IfModule include_module>
    Options +Includes +IncludesNoExec
</IfModule>

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    #CustomLog "logs/access_log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    CustomLog "logs/access_log" combined
	
	# LoadModule log_forensic_module modules/mod_log_forensic.so
	# LoadModule unique_id_module modules/mod_unique_id.so
	# ForensicLog logs/forensic_log
</IfModule>

<IfModule alias_module>
    #
    # Redirect: Allows you to tell clients about documents that used to 
    # exist in your server's namespace, but do not anymore. The client 
    # will make a new request for the document at its new location.
    # Example:
    # Redirect permanent /foo http://www.example.com/bar

    #
    # Alias: Maps web paths into filesystem paths and is used to
    # access content that does not live under the DocumentRoot.
    # Example:
    # Alias /webpath /full/filesystem/path
    #
    # If you include a trailing / on /webpath then the server will
    # require it to be present in the URL.  You will also likely
    # need to provide a <Directory> section to allow access to
    # the filesystem path.

    #
    # ScriptAlias: This controls which directories contain server scripts. 
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the target directory are treated as applications and
    # run by the server when requested rather than as documents sent to the
    # client.  The same rules about trailing "/" apply to ScriptAlias
    # directives as to Alias.
    #
    # ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
# <Directory "/var/www/cgi-bin">
#     AllowOverride None
#     Options None
#     Require all granted
# </Directory>

<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig /etc/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
	AddHandler server-parsed .html .htm
	
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml .xhtml .html .htm
</IfModule>

AddDefaultCharset UTF-8

<IfModule mime_magic_module>
    #
    # The mod_mime_magic module allows the server to use various hints from the
    # contents of the file itself to determine its type.  The MIMEMagicFile
    # directive tells the module where the hint definitions are located.
    #
    MIMEMagicFile conf/magic
</IfModule>

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# EnableMMAP and EnableSendfile: On systems that support it, 
# memory-mapping or the sendfile syscall may be used to deliver
# files.  This usually improves server performance, but must
# be turned off when serving from networked-mounted 
# filesystems or if support for these functions is otherwise
# broken on your system.
# Defaults if commented: EnableMMAP On, EnableSendfile Off
#
#EnableMMAP off
EnableSendfile on

# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf

Timeout 60
# the default Apache time-out value is 300 seconds, which can make
# the server subject to Slow Loris and DoS attacks; so this
# mitigates the problem with a lower timeout.

FileETag None
# unless set to None, the FileETag allows remote attackers to obtain
# sensitive information like inode number, multipart MIME boundary,
# and child process via the Etag header. Also a required fix for
# PCI compliance

TraceEnable off
# disallow cross-site tracing attacks [eg: via Telnet]

ServerTokens Prod
# changes the header to production only [ie: Apache, and nothing further]

ServerSignature Off
# removes version information from generated web pages

Header set X-XSS-Protection "1; mode=block"
# Cross Site Scripting (XSS) protection can be bypassed in many browsers, so this re-introduces it
# in the event of it being disabled

Header always append X-Frame-Options SAMEORIGIN   
# prevent clickjacking. Options include:
# DENY [will entirely prevent a page displaying in a frame or iframe]
# ALLOW-FROM uri [will allow a page to be displayed only on the specified origin]
# SAMEORIGIN [will allow a page to be displayed in a frame on the same origin as the page itself]

Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure
# mitigate most of the common Cross Site Scripting attacks using HttpOnly
# and Secure flag in a cookie, in order to avoid stolen or manipulated
# cookies and web application sessions.

#10

I’m sorry, I didn’t look closely at your previous problem when saying that you had found a very unusual bug. I was thinking about the ns2 part. The ns2 part is explained by your old configuration containing

because Certbot uses the ServerAlias to decide which names to obtain certificates for with --apache. So that isn’t as mysterious as I thought.

However, you have definitely still found a bug related to the failure to obtain a certificate with --apache now; the

is not expected behavior, and the files you provided should help in figuring this out.

Unfortunately, I’m about to go on vacation and several of my colleagues who work on Certbot are traveling for a Let’s Encrypt event. So we might not be able to get you a response quickly from the Certbot team.

I’ll bring this thread to my colleagues attention and hopefully someone from the Certbot project or another community forum member can help in figuring this out.

If you need a certificate in the meantime, you could also consider using --standalone or --webroot instead of --apache. Using --standalone would require stopping your Apache server temporarily during the certificate issuance process (and also during subsequent renewals), while using --webroot would require that there is a directory where you can place static files in order to have the web server serve them to the public at corresponding paths.


#11

Thanks for the further advice and update, - I do not mind waiting until you or your colleagues are able to take a further look at things, and will see if I can work something out in the meantime.

As it stands at the moment certbot --apache generates the following:

[root@localhost ~]# certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: insurgent.info
2: ns2.insurgent.info
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for insurgent.info
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. insurgent.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://insurgent.info/.well-known/acme-challenge/zutAfyahn1csRebA9rNeGbyInLcZopTKfPb5SNIOy9I: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: insurgent.info
   Type:   unauthorized
   Detail: Invalid response from
   http://insurgent.info/.well-known/acme-challenge/zutAfyahn1csRebA9rNeGbyInLcZopTKfPb5SNIOy9I:
   "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

#12

Problem now solved: there needs to be a virtual host directive in the ssl.conf file, eg:

<VirtualHost *:443>
  ServerAdmin info@some.info
  DocumentRoot "/var/www"
  ServerName insurgent.info
  SSLCertificateFile /etc/letsencrypt/live/insurgent.info/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/insurgent.info/privkey.pem
  Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

Note: the last three lines are added as part of the Certbot --apache process, so they will not be present when the VirtualHost directive is first added to ssl.conf.

Making two separate directives for IPv4 and IPv6 does not work and results in a broken configuration for IPv6 connections.

Also, it was necessary to remove all instances of ServerAlias and ServerName from the VirtualHost directives within the httpd.conf file, in addition to commenting-out the global ServerName value (ie: the default httpd.conf value).

These changes finally allowed Certbot --apache to install the certificates and I can now confirm that I have https:// working just as it should, with no browser complaints whatsoever, and no certificate exceptions required, in both Waterfox (a Firefox fork) and Internet Explorer. Thanks to all who made this possible and all on the forum who helped with my enquiry.


#13

For what it’s worth, I suspect what happened is that Certbot reconfigured the IPv4 virtual host to pass the challenge (because it was first), while Let’s Encrypt connected using IPv6 (because it prefers IPv6) and got the 404 Not Found error.


#14

Sounds possible, - I guess that dual-stack home servers are still a bit uncommon (in the UK, at least), added to which I am still dealing with some small glitches (referers, mainly, which insist on having a global ServerName which, in turn, messes-up the https://), and validators which can be funny about recognising https://; but I think that most things outside of the Ordnance Survey’s OpenSpace API are now tamed. Let me know if you need me to test any new configurations, and I will see what I can do for you.


#15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.