Missed Vhost on Port 80 (after Apache update), but it is there

My domain is: co2-avatar.com

I ran this command: certbot renew --dry-run

My web server is (include version): Apache 2.4.53

The operating system my web server runs on is (include version): CentOS 7 (I know)

$ certbot --version
certbot 1.11.0

Details:

I have updated Apache from 2.4.6 to 2.4.5x quite recently.

Now certbot renew --dry-run tells me that Apache is not listening on port 80

From /var/log/letsencrypt/letsencrypt.log

PluginError: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.

2022-04-27 13:00:18,010:ERROR:certbot._internal.renewal:All renewal attempts failed. The following certs could not be renewed:
2022-04-27 13:00:18,011:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/co2-avatar.com/fullchain.pem (failure)

But I have a <VirtualHost> in my apache config, which contains all requested domains as ServerName/ServerAlias. I have doublechecked with netstat that Apache is listening on port 80.

Here is the config file:

# redirect everything from http to https
<VirtualHost *:80>
    DocumentRoot "/var/www/html/"
    ServerName co2-avatar.com
    ServerAlias sustainable-data-platform.de
    ServerAlias co2-avatar.de
    ServerAlias co2-avatar.eu
    ServerAlias co2-avatar.org
    ServerAlias co2avatar.com
    ServerAlias co2avatar.de
    ServerAlias co2avatar.eu
    ServerAlias co2avatar.org
    ServerAlias git.sustainable-data-platform.org
    ServerAlias gitlab.sustainable-data-platform.org
    ServerAlias hp-cockpit.com
    ServerAlias hp-cockpit.de
    ServerAlias hp-cockpit.eu
    ServerAlias hp-cockpit.org
    ServerAlias hpcockpit.com
    ServerAlias hpcockpit.de
    ServerAlias hpcockpit.eu
    ServerAlias hpcockpit.org
    ServerAlias stop-fossil.de
    ServerAlias stop-fossil.org
    ServerAlias stopfossil.de
    ServerAlias stopfossil.org
    ServerAlias sustainable-building-platform.com
    ServerAlias sustainable-building-platform.de
    ServerAlias sustainable-building-platform.eu
    ServerAlias sustainable-building-platform.org
    ServerAlias sustainable-data-platform.com
    ServerAlias sustainable-data-platform.eu
    ServerAlias sustainable-data-platform.org
    ServerAlias sustainabledataplatform.com
    ServerAlias sustainabledataplatform.de
    ServerAlias sustainabledataplatform.eu
    ServerAlias sustainabledataplatform.org
    ServerAlias test.co2avatar.org
    ServerAlias test.hp-cockpit.org
    ServerAlias wp-cockpit.de
    ServerAlias wp-cockpit.eu
    ServerAlias wp-cockpit.org
    ServerAlias wpcockpit.eu
    ServerAlias wpcockpit.org

    <Directory "/var/www/html/">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
    ErrorLog "/var/log/httpd/webServer-error_log"
    CustomLog "/var/log/httpd/webServer-access_log" combined
    RewriteEngine on
    RewriteCond %{HTTPS} off
    RewriteCond %{REQUEST_URI} !\.well-known/acme-challenge
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

There might be something wrong with RewriteCond %{REQUEST_URI} !\.well-known/acme-challenge (which is from here), but how can I debug this? Are there more detailed log files from certbot or letsencrypt.

When run the renew or when I want to add some domains to my certificate, I indeed do not see any access in my Apache logfiles. So I am not sure, what I am doing wrong here and how to check, what actually happens on their side (Simulating renewal of an existing certificate for sustainable-data-platform.org and 40 more domains). Which is differently when I run curl -v -X GET http://co2-avatar.com/.well-known/acme-challenge on my machine. I get at least a 301 for this and an entry in the Apache logfile on my server.

Is there any other hint for using certbot with Apache 4.2.53?

1 Like

Please show the output of:

sudo certbot certificates --cert-name co2-avatar.com

And also please show the output of:

sudo apachectl -S

3 Likes
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following matching certs:
  Certificate Name: co2-avatar.com
    Serial Number: 349f7a16d60bb7a8f9768cf8997cf52a582
    Key Type: RSA
    Domains: sustainable-data-platform.org co2-avatar.com co2-avatar.de co2-avatar.eu co2-avatar.org co2avatar.com co2avatar.de co2avatar.eu co2avatar.org git.sustainable-data-platform.org gitlab.sustainable-data-platform.org hp-cockpit.com hp-cockpit.de hp-cockpit.eu hp-cockpit.org hpcockpit.com hpcockpit.de hpcockpit.eu hpcockpit.org stop-fossil.de stop-fossil.org stopfossil.de stopfossil.org sustainable-building-platform.com sustainable-building-platform.de sustainable-building-platform.eu sustainable-building-platform.org sustainable-data-platform.com sustainable-data-platform.de sustainable-data-platform.eu sustainabledataplatform.com sustainabledataplatform.de sustainabledataplatform.eu sustainabledataplatform.org test.co2avatar.org test.hp-cockpit.org wp-cockpit.de wp-cockpit.eu wp-cockpit.org wpcockpit.eu wpcockpit.org
    Expiry Date: 2022-06-22 13:28:03+00:00 (VALID: 54 days)
    Certificate Path: /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/co2-avatar.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

For VHosts

Key is

*:80                   is a NameVirtualHost
         default server co2-avatar.com (/etc/httpd/conf.d/000-vhost.conf:2)
         port 80 namevhost co2-avatar.com (/etc/httpd/conf.d/000-vhost.conf:2)
                 alias sustainable-data-platform.de
                 ...

Complete Output

$ sudo apachectl -S
VirtualHost configuration:
127.0.0.1:7010         ibo-php.localhost (/etc/httpd/conf.d/ibo-php-fcgi-prod.conf:5)
*:80                   is a NameVirtualHost
         default server co2-avatar.com (/etc/httpd/conf.d/000-vhost.conf:2)
         port 80 namevhost co2-avatar.com (/etc/httpd/conf.d/000-vhost.conf:2)
                 alias sustainable-data-platform.de
                 alias co2-avatar.de
                 alias co2-avatar.eu
                 alias co2-avatar.org
                 alias co2avatar.com
                 alias co2avatar.de
                 alias co2avatar.eu
                 alias co2avatar.org
                 alias git.sustainable-data-platform.org
                 alias gitlab.sustainable-data-platform.org
                 alias hp-cockpit.com
                 alias hp-cockpit.de
                 alias hp-cockpit.eu
                 alias hp-cockpit.org
                 alias hpcockpit.com
                 alias hpcockpit.de
                 alias hpcockpit.eu
                 alias hpcockpit.org
                 alias stop-fossil.de
                 alias stop-fossil.org
                 alias stopfossil.de
                 alias stopfossil.org
                 alias sustainable-building-platform.com
                 alias sustainable-building-platform.de
                 alias sustainable-building-platform.eu
                 alias sustainable-building-platform.org
                 alias sustainable-data-platform.com
                 alias sustainable-data-platform.eu
                 alias sustainable-data-platform.org
                 alias sustainabledataplatform.com
                 alias sustainabledataplatform.de
                 alias sustainabledataplatform.eu
                 alias sustainabledataplatform.org
                 alias test.co2avatar.org
                 alias test.hp-cockpit.org
                 alias wp-cockpit.de
                 alias wp-cockpit.eu
                 alias wp-cockpit.org
                 alias wpcockpit.eu
                 alias wpcockpit.org
         port 80 namevhost gitlab.sustainable-data-platform.org (/etc/httpd/conf.d/gitlab.conf:1)
                 alias sustainabledataplatform.org
*:443                  is a NameVirtualHost
         default server h2862201.stratoserver.net (/etc/httpd/conf.d/000-ssl.conf:57)
         port 443 namevhost h2862201.stratoserver.net (/etc/httpd/conf.d/000-ssl.conf:57)
         port 443 namevhost co2-avatar.com (/etc/httpd/conf.d/001-pages-php-fcgi.conf:2)
                 alias sustainable-data-platform.de
                 alias co2avatar.com
                 alias sustainabledataplatform.de
                 alias sustainable-building-platform.de
                 alias sustainable-data-platform.org
                 alias sustainabledataplatform.org
                 alias sustainable-building-platform.org
                 alias co2-avatar.org
                 alias sustainable-data-platform.com
                 alias sustainabledataplatform.com
                 alias sustainable-building-platform.com
                 alias co2avatar.com
                 alias co2-avatar.com
                 alias sustainable-data-platform.eu
                 alias sustainabledataplatform.eu
                 alias co2avatar.eu
                 alias co2-avatar.eu
                 alias co2avatar.de
                 alias co2-avatar.de
         port 443 namevhost wp-cockpit.de (/etc/httpd/conf.d/wp_cockpit_page.conf:4)
                 alias www.wp-cockpit.de
         port 443 namevhost test.co2avatar.org (/etc/httpd/conf.d/co2avatar.conf:7)
         port 443 namevhost co2avatar.org (/etc/httpd/conf.d/co2avatar.conf:66)
         port 443 namevhost hp-cockpit.org (/etc/httpd/conf.d/hpcockpit.conf:7)
                 alias hp-cockpit.com
                 alias hp-cockpit.de
                 alias hp-cockpit.eu
                 alias hp-cockpit.org
                 alias hpcockpit.com
                 alias hpcockpit.de
                 alias hpcockpit.org
         port 443 namevhost test.hp-cockpit.org (/etc/httpd/conf.d/hpcockpit.conf:72)
                 alias wpcockpit.eu
                 alias hpcockpit.eu
                 alias heatpumpcheck.de
         port 443 namevhost sustainable-building-platform.eu (/etc/httpd/conf.d/ec2-prod.conf:11)
         port 443 namevhost stop-fossil.de (/etc/httpd/conf.d/fossil.conf:7)
                 alias stop-fossil.org
                 alias stopfossil.de
                 alias stopfossil.org
         port 443 namevhost gitlab.sustainable-data-platform.org (/etc/httpd/conf.d/gitlab-le-ssl.conf:2)
ServerRoot: "/etc/httpd"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/etc/httpd/logs/error_log"
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex authdigest-client: using_defaults
Mutex lua-ivm-shm: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/etc/httpd/run/" mechanism=default 
Mutex mpm-accept: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex proxy-balancer-shm: using_defaults
PidFile: "/etc/httpd/run/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="apache" id=48
Group: name="apache" id=48
1 Like

I don't see anything weird. Could you perhaps share the letsencrypt.log file (contents or file) from the /var/log/letsencrypt/ directory?

3 Likes

certbot is probably looking in the wrong directory. try some of these options:

apache:
  Apache Web Server plugin (Please note that the default values of the
  Apache plugin options change depending on the operating system Certbot is
  run on.)

  --apache-enmod APACHE_ENMOD
                        Path to the Apache 'a2enmod' binary (default: None)
  --apache-dismod APACHE_DISMOD
                        Path to the Apache 'a2dismod' binary (default: None)
  --apache-le-vhost-ext APACHE_LE_VHOST_EXT
                        SSL vhost configuration extension (default: -le-
                        ssl.conf)
  --apache-server-root APACHE_SERVER_ROOT
                        Apache server root directory (default: /etc/apache2)
  --apache-vhost-root APACHE_VHOST_ROOT
                        Apache server VirtualHost configuration root (default:
                        None)
  --apache-logs-root APACHE_LOGS_ROOT
                        Apache server logs directory (default:
                        /var/log/apache2)
  --apache-challenge-location APACHE_CHALLENGE_LOCATION
                        Directory path for challenge configuration (default:
                        /etc/apache2)
  --apache-handle-modules APACHE_HANDLE_MODULES
                        Let installer handle enabling required modules for you
                        (Only Ubuntu/Debian currently) (default: False)
  --apache-handle-sites APACHE_HANDLE_SITES
                        Let installer handle enabling sites for you (Only
                        Ubuntu/Debian currently) (default: False)
  --apache-ctl APACHE_CTL
                        Full path to Apache control script (default:
                        apache2ctl)
  --apache-bin APACHE_BIN
                        Full path to apache2/httpd binary (default: None)

https://eff-certbot.readthedocs.io/en/stable/using.html#certbot-command-line-options

2 Likes

I am confused by the line * "expires": "2022-05-04T10:38:45Z"*, because certbot certificates says Expiry Date: 2022-06-22 13:28:03+00:00 (VALID: 54 days)

Is it possible having two certificates by some accident? I can see only one set of certificate files in our /etc/letsencrypt/live/co2-avatar.com.

Here we go: tail -150

# there are a lot of these blocks, because of the number of requested domains:

2022-04-29 15:28:05,037:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/2295905964:
{
  "protected": "eyJub25jZSI6ICIwMDAyczVESURnX2lTQjNXOElaLTNjak5HMzRYbk8xdVQwWk1zUTN0b2haWkZFNCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8yMjk1OTA1OTY0IiwgImtpZCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMjQ0MTI1NjgiLCAiYWxnIjogIlJTMjU2In0", 
  "payload": "", 
  "signature": "UK36jovu6qyYNKuZ1PE1brrmiCOQFsVc0nsTiJGwXGnSN0JHmEre26mOUBbfGKsUslV6NMbxUpGcovgGClDZtaa68BO8b-GM5l2I33-7ojt37GzmvFOWA-zrlAtsekwR_g-ReoO2SRrBlNgfkmDWvViPSy2_jL3Pa7YSA6WxyrV0eCWE2BoVk3VS8j9oFLkV3ljiQ-1hAi5gxY-WETGpigz5IeZTBPc1v5ZoEA8y_444wXKzYxkAzUxlKvf6hJABfNDUXRcETM9-nNo5zM4xBHsQR19D7JaLSYl3KcPk2kFyPWrR67e_4txScHDU1vtxCFY4Wv1K3gR20y82s1QO9g"
}
2022-04-29 15:28:05,197:DEBUG:urllib3.connectionpool:"POST /acme/authz-v3/2295905964 HTTP/1.1" 200 815
2022-04-29 15:28:05,197:DEBUG:acme.client:Received response:
HTTP 200
content-length: 815
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 24412568
date: Fri, 29 Apr 2022 13:28:05 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0002hzoacps9sOB9OA6E2iwVqFEKxgUvJBcB3LkNU2W9UBU

{
  "identifier": {
    "type": "dns",
    "value": "wp-cockpit.eu"
  },
  "status": "pending",
  "expires": "2022-05-04T10:38:45Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2295905964/6IelFw",
      "token": "wXL4bAGuHqTslGlvcufKGDIp8i8nfVH3fW8jpCTfYaw"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2295905964/Y4pAew",
      "token": "wXL4bAGuHqTslGlvcufKGDIp8i8nfVH3fW8jpCTfYaw"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2295905964/taw4uA",
      "token": "wXL4bAGuHqTslGlvcufKGDIp8i8nfVH3fW8jpCTfYaw"
    }
  ]
}
2022-04-29 15:28:05,198:DEBUG:acme.client:Storing nonce: 0002hzoacps9sOB9OA6E2iwVqFEKxgUvJBcB3LkNU2W9UBU
2022-04-29 15:28:05,198:DEBUG:acme.client:JWS payload:

2022-04-29 15:28:05,689:INFO:certbot._internal.auth_handler:Performing the following challenges:
2022-04-29 15:28:05,689:INFO:certbot._internal.auth_handler:http-01 challenge for co2-avatar.com
2022-04-29 15:28:05,690:INFO:certbot._internal.auth_handler:http-01 challenge for co2-avatar.de
2022-04-29 15:28:05,690:INFO:certbot._internal.auth_handler:http-01 challenge for co2-avatar.eu
2022-04-29 15:28:05,690:INFO:certbot._internal.auth_handler:http-01 challenge for co2-avatar.org
2022-04-29 15:28:05,690:INFO:certbot._internal.auth_handler:http-01 challenge for co2avatar.com
2022-04-29 15:28:05,691:INFO:certbot._internal.auth_handler:http-01 challenge for co2avatar.de
2022-04-29 15:28:05,691:INFO:certbot._internal.auth_handler:http-01 challenge for co2avatar.eu
2022-04-29 15:28:05,691:INFO:certbot._internal.auth_handler:http-01 challenge for co2avatar.org
2022-04-29 15:28:05,691:INFO:certbot._internal.auth_handler:http-01 challenge for git.sustainable-data-platform.org
2022-04-29 15:28:05,692:INFO:certbot._internal.auth_handler:http-01 challenge for gitlab.sustainable-data-platform.org
2022-04-29 15:28:05,692:INFO:certbot._internal.auth_handler:http-01 challenge for hp-cockpit.com
2022-04-29 15:28:05,692:INFO:certbot._internal.auth_handler:http-01 challenge for hp-cockpit.de
2022-04-29 15:28:05,692:INFO:certbot._internal.auth_handler:http-01 challenge for hp-cockpit.eu
2022-04-29 15:28:05,692:INFO:certbot._internal.auth_handler:http-01 challenge for hp-cockpit.org
2022-04-29 15:28:05,693:INFO:certbot._internal.auth_handler:http-01 challenge for hpcockpit.com
2022-04-29 15:28:05,693:INFO:certbot._internal.auth_handler:http-01 challenge for hpcockpit.de
2022-04-29 15:28:05,693:INFO:certbot._internal.auth_handler:http-01 challenge for hpcockpit.eu
2022-04-29 15:28:05,693:INFO:certbot._internal.auth_handler:http-01 challenge for hpcockpit.org
2022-04-29 15:28:05,693:INFO:certbot._internal.auth_handler:http-01 challenge for stop-fossil.de
2022-04-29 15:28:05,693:INFO:certbot._internal.auth_handler:http-01 challenge for stop-fossil.org
2022-04-29 15:28:05,694:INFO:certbot._internal.auth_handler:http-01 challenge for stopfossil.de
2022-04-29 15:28:05,694:INFO:certbot._internal.auth_handler:http-01 challenge for stopfossil.org
2022-04-29 15:28:05,694:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-building-platform.com
2022-04-29 15:28:05,694:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-building-platform.de
2022-04-29 15:28:05,694:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-building-platform.eu
2022-04-29 15:28:05,695:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-building-platform.org
2022-04-29 15:28:05,695:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-data-platform.com
2022-04-29 15:28:05,695:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-data-platform.de
2022-04-29 15:28:05,695:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-data-platform.eu
2022-04-29 15:28:05,695:INFO:certbot._internal.auth_handler:http-01 challenge for sustainable-data-platform.org
2022-04-29 15:28:05,695:INFO:certbot._internal.auth_handler:http-01 challenge for sustainabledataplatform.com
2022-04-29 15:28:05,696:INFO:certbot._internal.auth_handler:http-01 challenge for sustainabledataplatform.de
2022-04-29 15:28:05,696:INFO:certbot._internal.auth_handler:http-01 challenge for sustainabledataplatform.eu
2022-04-29 15:28:05,696:INFO:certbot._internal.auth_handler:http-01 challenge for sustainabledataplatform.org
2022-04-29 15:28:05,696:INFO:certbot._internal.auth_handler:http-01 challenge for test.co2avatar.org
2022-04-29 15:28:05,696:INFO:certbot._internal.auth_handler:http-01 challenge for test.hp-cockpit.org
2022-04-29 15:28:05,697:INFO:certbot._internal.auth_handler:http-01 challenge for wp-cockpit.de
2022-04-29 15:28:05,697:INFO:certbot._internal.auth_handler:http-01 challenge for wp-cockpit.eu
2022-04-29 15:28:05,697:INFO:certbot._internal.auth_handler:http-01 challenge for wp-cockpit.org
2022-04-29 15:28:05,697:INFO:certbot._internal.auth_handler:http-01 challenge for wpcockpit.eu
2022-04-29 15:28:05,697:INFO:certbot._internal.auth_handler:http-01 challenge for wpcockpit.org
2022-04-29 15:28:05,868:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 70, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/configurator.py", line 2425, in perform
    http_response = http_doer.perform()
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/http_01.py", line 76, in perform
    self._mod_config()
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/http_01.py", line 111, in _mod_config
    for vh in self._relevant_vhosts():
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/http_01.py", line 166, in _relevant_vhosts
    " {0}.".format(http01_port))
PluginError: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.

2022-04-29 15:28:05,869:DEBUG:certbot._internal.error_handler:Calling registered functions
2022-04-29 15:28:05,869:INFO:certbot._internal.auth_handler:Cleaning up challenges
2022-04-29 15:28:06,218:ERROR:certbot._internal.renewal:Failed to renew certificate co2-avatar.com with error: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.
2022-04-29 15:28:06,234:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 471, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1235, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 124, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 331, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 70, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/configurator.py", line 2425, in perform
    http_response = http_doer.perform()
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/http_01.py", line 76, in perform
    self._mod_config()
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/http_01.py", line 111, in _mod_config
    for vh in self._relevant_vhosts():
  File "/usr/lib/python2.7/site-packages/certbot_apache/_internal/http_01.py", line 166, in _relevant_vhosts
    " {0}.".format(http01_port))
PluginError: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.

2022-04-29 15:28:06,234:DEBUG:certbot.display.util:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-04-29 15:28:06,234:ERROR:certbot._internal.renewal:All simulated renewals failed. The following certificates could not be renewed:
2022-04-29 15:28:06,234:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/co2-avatar.com/fullchain.pem (failure)
2022-04-29 15:28:06,235:DEBUG:certbot.display.util:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-04-29 15:28:06,238:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==1.11.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1421, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1318, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 497, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)
2022-04-29 15:28:06,239:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)
2 Likes

That's just the expiry date of the authorization, not the certificate.

Looks like almost all hostnames have already been validated, but for some reason Certbot doesn't play well during the validation of wpcockpit.org. But why? I dunno :person_shrugging: Your configuration doesn't seem to be that weird.

Maybe @certbot-devs can chip in here.

3 Likes

Now I have tried sudo certbot renew --dry-run --apache --apache-server-root /etc/httpd (if this is the right syntax for httpd.

The result is the same and the folders did not change at all, but the same command has worked perfectly before updating Apache (2.4.6 -> 2.4.53).

1 Like

Thanks anyway!

2 Likes

Due to the variety of different layouts for Apache httpd in different distros the way our Apache plugin works is it detects the OS it's running on and then tries to interact with Apache using the conventions of that OS. For instance, here is our Apache code for 1.11.0 for CentOS. You can see the defaults for the configurable values Certbot will use on your system by running certbot --help apache. My guess is the layout of your new Apache installation is slightly different and giving Certbot trouble.

To test that theory:

  1. How did you install Apache 2.4.53 on CentOS 7?
  2. Does the output of certbot --help apache match what you expect for your new installation?
  3. Does /etc/sysconfig/httpd exist on your system and is it used by your new Apache installation?

If the answers to (3) are yes and no respectively, I think it's worth trying temporarily moving that file somewhere else and trying to renew your cert again to see if solves the problem. As described in the links above, that file is provided by the default CentOS packages and we need to parse it to work with Apache provided by your OS, but if it's for your old Apache installation, it could be confusing Certbot.

7 Likes

Well, there is one active line is this file: LANG=C, moving it to some tmp folder did not help.

Updating Apache on CentOS 7 (I know) is a nightmare. I have used package epel-release and fetched the repo file (.repo) from codeit (https://repo.codeit.guru/codeit.el7.repo), then updated httpd with yum update httpd. So I have

$ apachectl -v
Server version: Apache/2.4.53 (codeit)
Server built:   Mar 14 2022 11:48:19

ad 2)

After scanning the options for CentOS I was interested in --apache-le-vhost-ext, because I have multiple files for mulitple Vhosts. I did not see this option in use in the python file, which you have linked here, but which file would be the most important one? I have tried to use the extension for the basic SSL configuration file, but also other Vhost files. Nothing changed the result.

So I am still stuck here.

1 Like

I tried to reproduce this by creating a new CentOS 7 server, installing everything, and then adding your /etc/httpd/conf.d/000-vhost.conf file. At least with this (very) simplified setup, I wasn't able to reproduce the problem and Certbot was able to find the port 80 vhosts. By getting more details about your setup such as the contents of some of your other httpd config files, we may be able to reproduce and debug this, but I have another option for you.

Since you've already set up an exception for .well-known/acme-challenge, you can probably use our webroot plugin to obtain certificates while continuing to use our Apache plugin to install them.

A command like:

sudo certbot renew --cert-name co2-avatar.com --dry-run --authenticator webroot --webroot-path /var/www/html --installer apache

would test running Certbot in this way. If it works, you could cause this certificate to be renewed this way now and in the future by running the same command without --dry-run.

4 Likes

Using the webroot options does work!

With --dry-run

Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/co2-avatar.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all simulated renewals succeeded: 
  /etc/letsencrypt/live/co2-avatar.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

OK, I think, I can also do a sudo certbot certonly --cert-name co2-avatar.com -d sustainable-data-platform.org ... with the webroot option (purpose: adding domains to the certificate)?

Then, just because I'd like to get it: what does cerbot do with --apache, but without webroot? I understand from reading the docs that webroot means using a temporary file for each domain in the web-root-folder. But why did the verification work without this process before?

Thanks a lot!

4 Likes

If you want to add a name to a certificate you will need to expand it or relist all the names + the new one.

It would use Apache for authentication and also update the Apache config with TLS server block.

3 Likes

Yes that will work, but you may want to remove certonly from the command. The reason for this is Certbot will install/configure the certificates with Apache which may need to be done if you've created new vhosts that should use the certificate.

Correct. Certbot with --apache is equivalent to --authenticator apache --installer apache. We define these terms in the context of Certbot a bit here.

When our Apache plugin is used at the authenticator, it does something quite similar to webroot, but parses and temporarily modifies your Apache config files to make that happen.

That's a mystery. In my previous post I described how I attempted to recreate the problem but was unable to do so. Something changed in the environment on your system which caused Certbot to fail to parse your Apache config files correctly. I think it'd be nice to track it down, but it's going to be a little tricky and --authenticator webroot --installer apache shouldn't have any downsides if it works for you.

4 Likes

Thanks again.

Something changed in the environment on your system which caused Certbot to fail to parse your Apache config files correctly.

This might be the case of course, but I am not familiar with Apache internals. Therefore I cannot add much to this. I did this update in order to change some configuration and for using some features, which were not available in v2.4.6. So I changed config files and renamed them, this could also cause the problems.

The main SSL VHost file for examle was named vhost-le-ssl.conf and looked like this:

<IfModule mod_ssl.c>

# Listen 443
<VirtualHost *:443>
    DocumentRoot "/var/www/html/"
    ServerName co2-avatar.com
    ServerAlias sustainable-data-platform.de
    ServerAlias co2avatar.com
    ServerAlias sustainabledataplatform.de
    ServerAlias sustainable-building-platform.de
    ServerAlias sustainable-data-platform.org
    ServerAlias sustainabledataplatform.org
    ServerAlias sustainable-building-platform.org
    # ServerAlias co2avatar.org
    ServerAlias co2-avatar.org
    ServerAlias sustainable-data-platform.com
    ServerAlias sustainabledataplatform.com
    ServerAlias sustainable-building-platform.com
    ServerAlias co2avatar.com
    ServerAlias co2-avatar.com
    ServerAlias sustainable-data-platform.eu
    ServerAlias sustainabledataplatform.eu
    ServerAlias sustainable-building-platform.eu
    ServerAlias co2avatar.eu
    ServerAlias co2-avatar.eu
    ServerAlias co2avatar.de
    ServerAlias co2-avatar.de
    ServerAlias stopfossil.de
    ServerAlias stopfossil.org

    <Directory "/var/www/html/">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    ErrorLog "/var/log/httpd/webServer-error_log"
    CustomLog "/var/log/httpd/webServer-access_log" combined

    SSLEngine on
    # SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
    
    SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/cert.pem
    SSLCertificateChainFile /etc/letsencrypt/live/co2-avatar.com/chain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/co2-avatar.com/privkey.pem
    
    Include /etc/letsencrypt/options-ssl-apache.conf

    #SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
    #SSLCertificateKeyFile /etc/letsencrypt/live/co2-avatar.com/privkey.pem
    # SSLCertificateChainFile /etc/letsencrypt/live/co2-avatar.com/chain.pem
</VirtualHost>
</IfModule>

Now it is 001-pages-php-fcgi.conf:

# actually main wordpress site, community and collaboration are in /var/www/html
<VirtualHost *:443>
    DocumentRoot "/var/www/html/"
    ServerName co2-avatar.com
    ServerAlias sustainable-data-platform.de
    ServerAlias co2avatar.com
    ServerAlias sustainabledataplatform.de
    ServerAlias sustainable-building-platform.de
    ServerAlias sustainable-data-platform.org
    ServerAlias sustainabledataplatform.org
    ServerAlias sustainable-building-platform.org
    # ServerAlias co2avatar.org
    ServerAlias co2-avatar.org
    ServerAlias sustainable-data-platform.com
    ServerAlias sustainabledataplatform.com
    ServerAlias sustainable-building-platform.com
    ServerAlias co2avatar.com
    ServerAlias co2-avatar.com
    ServerAlias sustainable-data-platform.eu
    ServerAlias sustainabledataplatform.eu
    ServerAlias sustainable-building-platform.eu
    ServerAlias co2avatar.eu
    ServerAlias co2-avatar.eu
    ServerAlias co2avatar.de
    ServerAlias co2-avatar.de
    ServerAlias stopfossil.de
    ServerAlias stopfossil.org

    Header always set Access-Control-Allow-Origin "*"
    Header always set Access-Control-Allow-Headers "Authorization, x-elgg-ajax-api, origin, x-requested-with, content-type"
    Header always set Access-Control-Allow-Methods "GET, POST, PUT, OPTIONS, DELETE"
    Header always set Access-Control-Allow-Credentials "true"
    # Header always set Access-Control-Expose-Headers "Content-Security-Policy, Location"
    Header set Access-Control-Max-Age "600"

    RewriteEngine On
    # RewriteCond %{REQUEST_METHOD} OPTIONS
    # RewriteRule ^(.*)$ $1 [R=200,L]


    # ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9000/var/www/html/$1.
    DirectoryIndex index.php
    <FilesMatch "\.php$">
        SetHandler proxy:fcgi://localhost:9000 
        # SetHandler "proxy:unix:/run/php/72php-fpm.sock|fcgi://co2avatar.com"
    </FilesMatch>

    # RewriteCond %{HTTPS} !=on
    # RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

    # RewriteCond %{HTTP_HOST} ^(www\.)?sustainable-data-platform.de$ [NC]
    # RewriteCond %{HTTPS} off
    # RewriteRule ^ https://sustainable-data-platform.de%{REQUEST_URI} [NC,L,R]
    # add www on ssl
    # RewriteCond %{HTTPS} on
    # RewriteCond %{HTTP_HOST} ^www\.
    # RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [NC,L,R]

    # RewriteCond %{REQUEST_URI} ^/community/ [NC]
    # RewriteRule . /maintenance/maintenance-simple.html [END]

    # RewriteCond %{REQUEST_URI} ^/community-test/(.) [NC]
    # RewriteRule https://%{SERVER_NAME}%/community/$1 [END,NE,R=permanent]
    
    <Directory "/var/www/html/">
        # CGIPassAuth On
        # SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    ErrorLog "/var/log/httpd/webServer-error_log"
    CustomLog "/var/log/httpd/webServer-access_log" combined

    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
    # SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/cert.pem
    # SSLCertificateChainFile /etc/letsencrypt/live/co2-avatar.com/chain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/co2-avatar.com/privkey.pem
    
    Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

One change might be interesting: the way of referencing certificate files:

In Apache v2.4.6 you need to use all three files: cert, chain, privkey. Now (>v2.4.??) you only need fullchain and privkey.

The problem might be on my side, because I might have missed some places for changing this and doing this coherently. I'll check this.

Update: We are using fullchain and privkey references only.

1 Like

Since 2.4.8 :wink: That's the version that SSLCertificateChainFile became deprecated in favour of using fullchain.pem in SSLCertificateFile. See also the documentation.

4 Likes

2.2 needed the three directives, it just did not support fullchain in SSLCertificateFile

After, fullchain became supported, preferred, and SSLCertificateChainFile was deprecated.

2 Likes

Wow this is a tricky one. I added 01-pages-php-fcgi.conf to my test server and I still wasn't able to reproduce the problem. It's not clear to me what would be different between our two setups that would cause you to hit the issue and me not to.

If I'm able to get instructions on how to reproduce this, I'll look into it further, but I'm glad I was able to give you a workaround that works for you.

3 Likes