Cannot renew, Apache, linux

Help
1

My domain is:
xbiologix.net, xbiologix.info
I ran this command:
/usr/bin/certbot --dry-run renew >>/var/log/letsencrypt/renew.log
It produced this output:
Failed to renew certificate issues.xbiologix.net with error: Some challenges have failed.
Failed to renew certificate xbiologix.info with error: Some challenges have failed.
Failed to renew certificate xbiologix.net with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.
All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/issues.xbiologix.net/fullchain.pem (failure)
/etc/letsencrypt/live/xbiologix.info/fullchain.pem (failure)
/etc/letsencrypt/live/xbiologix.net/fullchain.pem (failure)
3 renew failure(s), 0 parse failure(s)
My web server is (include version):
apache 2.4.58
The operating system my web server runs on is (include version):
ubuntu 24.04.1
My hosting provider, if applicable, is:
google
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
2.9.0

After running the above certbot command, if apache was NOT running already, there are 3 apache2 processes running:

# ps ax | grep apache
1930765 ?        Ssl    0:00 /usr/sbin/apache2 -k graceful
1930883 ?        Sl     0:00 /usr/sbin/apache2 -k graceful
1930885 ?        Sl     0:00 /usr/sbin/apache2 -k graceful

I'm pretty confused about what's going on. I had http access blocked, but turned it on (makes no difference). All dns records are pinging ok.
The certs are for the following (from the letsencrypt log):

  xbiologix.net
    www.xbiologix.net
    issues.xbiologix.net
    issues-test.xbiologix.net
    xblgx-ops.xbiologix.net
  xbiologix.info

However, /etc/letsencrypt/archive only has subdirectories for:

  xbiologix.net
  xbiologix.info
  issues.xbiologix.net

which seems pretty weird. The entries in issues.xbiologix.net are one day older than those in xbiologix.net, so may be the result of an error when adding that cert.
If I run the command with "-vv", I see

Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 1137
Received response:
HTTP 200
Server: nginx
Date: Wed, 19 Nov 2025 20:59:08 GMT
Content-Type: application/json
Content-Length: 1137
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

followed by POST requests with 200 and 201 responses, then:

Performing the following challenges:
dns-01 challenge for issues-test.xbiologix.net
dns-01 challenge for issues.xbiologix.net
dns-01 challenge for xblgx-ops.xbiologix.net
Running manual-auth-hook command: /etc/letsencrypt/acme-dns-auth.py
Running manual-auth-hook command: /etc/letsencrypt/acme-dns-auth.py
Running manual-auth-hook command: /etc/letsencrypt/acme-dns-auth.py

followed by more posts with 200 responses and then:

Storing nonce: K_WcMXRqbxmA926otqM7Py05KM2gkV-hUJ5BZjN6w5HFG_Xaxh8
Challenge failed for domain issues-test.xbiologix.net
Challenge failed for domain issues.xbiologix.net
Challenge failed for domain xblgx-ops.xbiologix.net
dns-01 challenge for issues-test.xbiologix.net
dns-01 challenge for issues.xbiologix.net
dns-01 challenge for xblgx-ops.xbiologix.net
Notifying user: 
Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
  Domain: issues-test.xbiologix.net
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.issues-test.xbiologix.net - check that a DNS record exists for this domain

  Domain: issues.xbiologix.net
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.issues.xbiologix.net - check that a DNS record exists for this domain

  Domain: xblgx-ops.xbiologix.net
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.xblgx-ops.xbiologix.net - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to "certbot --help manual" and the Certbot User Guide.

Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2

Oh my, there is quite a lot to "unpack" here. Before we even try to untangle that I have some questions for you. These will help guide the solution(s).

Are you at least a moderately skilled Apache administrator?

Do you use these certificates anywhere other than Apache?

Why did you start using acme-dns and is that necessary?

3 Likes
3

Thanks for the reply.
Yes, moderately skilled Apache administrator.
No, certs are not used anywhere else.
I don't know why we started using acme-dns, or that we even were. I've always just run certbot renew either manually or as a cron job. I think if I changed anything it was a result of a notice from letsencrypt about stuff no longer going to work the way it used to (a few months ago). I have no record of changing anything regarding use of certbot but that doesn't mean I didn't, although I try to save old versions of things along with dates. How would I renew without using acme-dns? And where would I be specifying to use it?

We have, however, modifed our apache config in the past two months as part of a hardening exercise. That involved introducing new virtual hosts and adding them to the certs (listed above). I think I've backed out most of the hardening but left the new hosts in.

4

Okay. That's helpful. I recommend stop using Certbot and instead use the ACME Client mod_md that is built in to Apache.

There is some one-time setup and then you place config settings in the VirtualHost to define the certs. There is no coordination needed with an "outside" ACME Client (like Certbot). It is really pretty easy.

mod_md even supports the TLS-ALPN challenge which occurs over port 443.

The Apache docs for it are: mod_md - Apache HTTP Server Version 2.4

The github for mod_md is here: GitHub - icing/mod_md: Let's Encrypt (ACME) in Apache httpd
There are some better examples and how-to in the github but the Apache docs are helpful too especially for the one-time setup.

Once your Apache config only references the mod_md settings you should run sudo certbot --delete to remove the certs that it manages. You don't want to continuously be trying to renew certs unnecessarily. You could delete Certbot itself.

I think this approach will get your system running quicker than fixing your Certbot setup. And, will be easier to maintain and thus more reliable going forward.

4 Likes
5

Thanks. Maybe making progress. I am trying to switch to using mod_md.
One apache server, 2 domains. Comments and irrelevent (I think) stuff omitted)

xbiologix.info (file md1.conf):

<IfModule mod_ssl.c>
MDomain xbiologix.info
<VirtualHost *:443>
    ServerName www.xbiologix.info
    ServerAlias xbiologix.info
    DocumentRoot /srv/info/stuff
    MDCertificateAgreement accepted
    SSLEngine on
    <Directory /srv/info/stuff/>
        Require all granted
        Options FollowSymLinks
    </Directory>
</VirtualHost>
</IfModule>

xbiologix.net (file md2.conf):

<IfModule mod_ssl.c>
MDomain xbiologix.net
<IfModule mod_headers.c>
    Header always set Strict-Transport-Security "max-age=0"
</IfModule>
<VirtualHost *:443>
    ServerName issues.xbiologix.net
    ServerAlias www.xbiologix.net
    ServerAlias xbiologix.net
    DocumentRoot /srv/net/stuff
    MDCertificateAgreement accepted
   SSLEngine on
   <Directory /srv/net/stuff/>
        Require all denied
    </Directory>
    <Directory /srv/net/stuff/html>
        Require all granted
        Options FollowSymLinks
  </Directory>
</VirtualHost>
<VirtualHost *:443>
    ServerName issues-test.xbiologix.net
    DocumentRoot /srv/net/stuff
    MDCertificateAgreement accepted
   SSLEngine on
    <Directory /srv/net/stuff/>
        Require all denied
    </Directory>
    <Directory /srv/net/stuff/html2>
        Require all granted
        Options FollowSymLinks
    </Directory>
<VirtualHost *:443>
    ServerName xblgx-ops.xbiologix.net
    ServerAlias xbiologix.net
    DocumentRoot /srv/net
    MDCertificateAgreement accepted
    SSLEngine on
    <Directory /srv/stuff>
        Require all granted
        AllowOverride None
        Options -Indexes -FollowSymLinks
    </Directory>
</VirtualHost>
</IfModule>

The error log for xbiologix.info shows:

[Fri Nov 21 04:40:26.008642 2025] [ssl:warn] ... Init: www.xbiologix.info:443 will respond with '503 Service Unavailable' for now. There are no SSL certificates configured and no other module contributed any.
[Fri Nov 21 04:40:26.012001 2025] [ssl:warn] ... www.xbiologix.info:443:0 server certificate does NOT include an ID which matches the server name

I don't understand that -- the hostname is issues.xbiologix.net, but there are DNS records pointing xbiologix.info to the same IP. Note that this is served by google, so a reverse IP (dig -x) will give you something at googleusercontent.com

The error log for xbiologix.net shows:

[Fri Nov 21 04:44:27.237195 2025] [md:error] ... (22)Invalid argument: md[xbiologix.info] while[Creating new ACME account for xbiologix.info] detail[the CA requires you to accept the terms-of-service as specified in <https://letsencrypt.org/documents/LE-SA-v1.6-August-18-2025.pdf>. Please read the document that you find at that URL and, if you agree to the conditions, configure "MDCertificateAgreement accepted" in your Apache. Then (graceful) restart the server to activate.]
[Fri Nov 21 04:44:27.238783 2025] [md:error] ... (22)Invalid argument: AH10056: processing xbiologix.info: the CA requires you to accept the terms-of-service as specified in <https://letsencrypt.org/documents/LE-SA-v1.6-August-18-2025.pdf>. Please read the document that you find at that URL and, if you agree to the conditions, configure "MDCertificateAgreement accepted" in your Apache. Then (graceful) restart the server to activate.

I don't understand the xbiologix.net error either -- each of the VirtualHost(s) in the conf file has the statement "MDCertificateAgreement accepted", and the error refers to xbiologix.info, not xbiologix.net.

6

Thought maybe there could only be one MDomain command so put it in apache2.conf before the sites are included

MDomain xbiologix.net, xbiologix.info

but it didn't change the errors. Also got rid of all but the first xbiologix.net VirtualHost.

7

You need to add this to config file to acknowledge you accept Let's Encrypt's Terms of Use.

2 Likes
8

Below are comments as you originally posted the VHosts. First, let's get better terms.

You say ...

But, each ServerName and ServerAlias is a domain name. You have 7 domain names (ignoring the duplicate I describe below).

No, that's not a restriction. There is flexibility but, for now, I suggest using one MDomain statement before each port 443 VirtualHost. You can name either the ServerName or one of the ServerAlias domain names.

And, don't put it in the apache.conf. Keep it with the VHost config file. It will be easier to manage this way

<VirtualHost *:443>
    ServerName issues-test.xbiologix.net

This VHost is missing an ending </VirtualHost> - perhaps just a copy/paste problem but review to be sure.

It is also missing an MDomain statement for its ServerName

<VirtualHost *:443>
    ServerName xblgx-ops.xbiologix.net
    ServerAlias xbiologix.net

This VHost ServerAlias names the same domain name as you used in a prior VHost for port 443. This is not valid.

A domain name must be unique among all VHost for the same port. Actually, without mod_md Apache will not give an error but it won't process requests for that name as you expect. With mod_md you should keep them unique as they should be.

Also add an MDomain for its unique domain name

I would clean up all this and show any errors along with output of:

sudo apache2ctl -t -D DUMP_VHOSTS
4 Likes
9

Thanks for the help.
The missing /VirtualHost was a cut and paste; it’s there.
There are actually 4 files, 1 each for port 80 and 1 each for port 443. Normally the port 80s do redirects to 443, but I've temporarily changed them to allow straight http access. If only the port 80s are enabled, things work fine. I've temporarily paired things down to the 2 port 80s and one of the port 443s. If I activate those 2 and only the xbiologix.net on port 443, I get a temporary cert for xbiologix.net. So far so good. But it never turns into an official cert. Isn't it supposed to do that auto-magically after some relatively short interval?

Not sure I understand what you’re saying. If I have

file1.conf:
  MDomain xbiologix.net
  <VirtualHost *:443>
    ServerName issues.xbiologix.net
    ServerAlias  www.xbiologix.net
    ServerAlias  xbiologix.net
    ...
  </VirtualHost>
  <VirtualHost *:443>
    ServerName xxx.xbiologix.net
    ...
  </VirtualHost>
file2.conf:
  MDomain xbiologix.info
  <VirtualHost *:443>
    ServerName www.xbiologix.info
    ServerAlias  xbiologix.info
    ...
  </VirtualHost>

Is that legal? Both xbiologix.info and xbiologix.net are on the same port.
Or do I need to omit the “ServerAlias xbiologix.info” from the last one?

Is the order in which apache / mod_md processes the VirtualHost blocks important?
Since these are in separate files, I have no clear control of the order of processing.

Also, are the temporary self-signed certs and the permanent certs stored locally by apache / mod_md? I couldn't find anything in /var.

@Patryk if you check post #5 there is an MDCertificateAgreement accepted in each VirtualHost.

10

Forgot to add the dump. After a few minutes and "apache2ctl -k graceful":

# apache2ctl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443                  issues.xbiologix.net (/etc/apache2/sites-enabled/xblgx-ssl.conf:56)
*:80                   is a NameVirtualHost
         default server www.xbiologix.info (/etc/apache2/sites-enabled/xblgx-info.conf:1)
         port 80 namevhost www.xbiologix.info (/etc/apache2/sites-enabled/xblgx-info.conf:1)
                 alias xbiologix.info
         port 80 namevhost www.xbiologix.net (/etc/apache2/sites-enabled/xblgx.conf:3)
                 alias xbiologix.net
11

One, that isn't what you showed earlier. You had xbiologix.net as a ServerAlias for two different VirtualHost both using port 443. That isn't legal.

In your file1.conf you should have an MDomain for xxx.biologix.net right before the VirtualHost for that.

I think some of your issues is that you think there are two domain names but that's not the case. You have two "registered" domain names (things you purchased) but each subdomain is also a domain name (or hostname). They are distinct. For these purposes the registered domain does not have special significance.

Would you post the contents of that config file?

3 Likes
12

correct, that was my error, the result of a tweak to try to get things working. Since fixed but in the current context not relevant because for now only one VirtualHost in that file.

Thanks, will try that once the single VirtualHost is working.

Ah... thanks.
The single :443 config file stripped of commented out stuff:

LoadModule passenger_module /path/to/mod_passenger.so
<IfModule mod_passenger.c>
  PassengerRoot and PassengerDefaultRuby paths here
</IfModule>
<IfModule mod_ssl.c>
MDomain xbiologix.net
ServerAdmin admin@someplace.tld
    ExtFilterDefine md-pandoc-to-html ...
    ExtFilterDefine md-multimarkdown-to-html ...
<IfModule mod_headers.c>
    Header always set Strict-Transport-Security "max-age=0"
</IfModule>
<VirtualHost *:443>
    ServerName issues.xbiologix.net
    ServerAlias www.xbiologix.net
    ServerAlias xbiologix.net
    DocumentRoot /srv/net/stuff
    MDContactEmail admin@someplace.tld
    MDCertificateAgreement accepted
    SSLEngine on
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log vhost_combined
    <Directory /srv/net/stuff/>
        Require all denied
    </Directory>
    <Directory /srv/net/stuff/html>
        Require all granted
        Options FollowSymLinks
       Passenger-related-stuff-here
    </Directory>
    <Location /srv/net/svn-stuff>
      ... svn related stuff
    </Location>
</VirtualHost>
</IfModule>
13

I don't see a hostname for issues.xbiologix.net in your HTTP list. Apache will handle those in the default server, which may be ok, but probably not for .net domain requests. You should really have each domain name defined in HTTP (which later redirect to HTTPS).

Yes, it should be very quick in normal situations. What does the Apache error log say?

Given all your changes you could try reset mod_md. See the File Storage notes here: GitHub - icing/mod_md: Let's Encrypt (ACME) in Apache httpd

Also this Starting Over section: GitHub - icing/mod_md: Let's Encrypt (ACME) in Apache httpd

Delete any suspect cert info and restart Apache. Do actual restart though not just graceful.

3 Likes
14

I was just reviewing these and those are supposed to be in the "server" context. Which means in the Apache2.conf not inside a VirtualHost. or, alternate config file but not inside VirtualHost

4 Likes
15

Thanks. Not sure I understand that. If I only want known clients from known origins which only connect via https to use that host, why would I need to specify it in the http VirtualHost?

After removing md dir, restart, 10 mins or so:

[Fri Nov 21 19:46:25.455817 2025] [md:error] ... (22)Invalid argument: AH10056: processing xbiologix.net: the CA requires you to accept the terms-of-service as specified in <https://letsencrypt.org/documents/LE-SA-v1.6-August-18-2025.pdf>. Please read the document that you find at that URL and, if you agree to the conditions, configure "MDCertificateAgreement accepted" in your Apache. Then (graceful) restart the server to activate.

Aargh. RTFM completely. missed that, thanks

I stopped apache, removed /etc/apache2/md, and restarted apache.
Originally, I think something was in both .../md/staging and .../md/domains
but after "apache2ctl -k graceful" there is only stuff in domains; I still have only a temporary cert:

# ls -lt domains/xbiologix.net
-rw------- 1 root root  520 Nov 21 19:32 md.json
-rw------- 1 root root 1168 Nov 21 19:32 fallback-pubcert.pem
-rw------- 1 root root 1704 Nov 21 19:32 fallback-privkey.pem
root@issues:/etc/apache2/md# ls -lt staging
total 0

/etc/apache2/md/staging/xbiologix.net/job.json has entries (funny order, most recent first) still saying to read the terms-of-service and add MDCertificateAgreement statement.
/etc/apache2/md/staging/xbiologix.net/md.json:

  "name": "xbiologix.net",
  "domains": [
    "xbiologix.net",
    "issues.xbiologix.net",
    "www.xbiologix.net"
  ],
  "contacts": [
    "mailto:admin@someplace.tld"   (not what's actually there)
  ],
  "transitive": 1,
  "ca": {    "proto": "ACME",    "url": "https://acme-v02.api.letsencrypt.org/directory",    "urls": [
      "https://acme-v02.api.letsencrypt.org/directory"
    ]
  },
  "state": 1,
  "state-descr": "certificate(rsa) is missing",
  "renew-mode": 1,
  "renew-window": "33%",
  "warn-window": "10%",  "must-staple": false,  "proto": {    "acme-tls/1": []
  },
  "stapling": false

Do I need to tell LetsEncrypt to get rid of the old certs even though they've expired? They explicitly say not to revoke them.

16

Didn't see that, will fix.

17

Well, any HTTP request inbound to Apache will get processed by your Default Host for port 80 anyway. Why not have the ServerName and ServerAlias in HTTP VHost match the equivalent HTTPS VHost?

No.

When I suggested starting mod_md over I thought perhaps there was just a mis-matched set of domains / configs causing a problem. But, I have since seen your error of having MD config settings in the VHost when they belong in the server context.

I am pretty sure once you sort that you will start to see success.

3 Likes
18

The original thought was to not expose the name somehow, although in retrospect that doesn't really make sense.

Hooray! Stopping apache, removing the md dir, moving MDContactEmail and MDCertificateAgreement to apache2.conf, restarting apache and then a graceful restart (not sure the graceful restart was required) did the trick.
Thank you! Will try to incrementally put things back together now, starting with adding the other domain.

2 Likes
19

Well, a restart now (aka systemctl restart) is harsh but "fresher". A graceful (aka systemctl reload) is usually all that is needed when applying Apache config changes. When weird stuff happens I like to restart. In the end we learned nothing weird was happening but at the time ... :slight_smile:

Once this is working you'll want to make a cronjob to do a graceful reload once / day.

This is required to pickup any new cert that mod_md acquired that day.

3 Likes
20

Thanks.
New cert being one it acquired as a result of a renew attempt?
Is there any way to ask md to reload without a restart?
Reason I'm asking is some of these sites require logging into apache, not just an app running on apache, and users leave their browsers up for days. Depending on the situation, that may (can't remember the particulars) force another login to apache, even if the app caches.