How to troubleshoot certbot renew timer not working correctly

[grvvy]

It's a bit complicated of a setup that someone else configured and documented. The gist of it is ElasticIP -> EC2 squid proxy -> OpenVPN -> local machine that runs the web server (Jenkins).

Certbot and it's timers live on the AWS EC2 instance and I'm able to login to a root shell there and check. I know very little about sysadmin and networking but I was able to find where the timers are listed on the EC2 instance but I'm not sure how to find (if there exists) a log of previous attempts to renew the cert via timer, or what the parameters of the actual command are being run on a timer.

Manually stopping squid, regenerating the cert, and starting squid works fine.

I'm also unsure how the EC2 instance is running such a new version of certbot when I haven't run any updates on the system myself in a while.

My domain is:
https://jenkins.greatscottgadgets.com

I ran this command:
systemctl list-timers

It produced this output:

NEXT                        LEFT         LAST                        PASSED        UNIT                         ACTIVATES                     
          
Wed 2026-02-04 05:11:00 UTC 8h left      n/a                         n/a           snap.certbot.renew.timer     snap.certbot.renew.service 

$ cat snap.certbot.renew.service
[Unit]
# Auto-generated, DO NOT EDIT
Description=Service for snap application certbot.renew
Requires=snap-certbot-5361.mount
Wants=network.target
After=snap-certbot-5361.mount network.target snapd.apparmor.service
X-Snappy=yes

[Service]
EnvironmentFile=-/etc/environment
ExecStart=/usr/bin/snap run --timer="00:00~24:00/2" certbot.renew
SyslogIdentifier=certbot.renew
Restart=no
WorkingDirectory=/var/snap/certbot/5361
TimeoutStopSec=30
Type=oneshot

$ cat snap.certbot.renew.timer
[Unit]
# Auto-generated, DO NOT EDIT
Description=Timer renew for snap application certbot.renew
Requires=snap-certbot-5361.mount
After=snap-certbot-5361.mount
X-Snappy=yes

[Timer]
Unit=snap.certbot.renew.service
OnCalendar=*-*-* 05:11
OnCalendar=*-*-* 20:35

[Install]
WantedBy=timers.target

My web server is (include version):
Jenkins?

The operating system my web server runs on is (include version):
Ubuntu 20.04.6 LTS

My hosting provider, if applicable, is:
AWS?

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
5.3.0

[grvvy]

Sorry, I'm new here and not sure how to edit the post.

Manually testing automatic renewal shows a failure to renew due to another process using the same port (although the url it complains about is not the one I use to access the site, but it is one included in the configuration from the previous person). I'm guessing that if I stop squid and retry it then it might work, and perhaps that's what's causing issues with the renewal but I could use some more educated opinions before trying that.

$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/greatscottgadgets.schneiderlab.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Account registered.
Simulating renewal of an existing certificate for jenkins.greatscottgadgets.com and greatscottgadgets.schneiderlab.com
Failed to renew certificate greatscottgadgets.schneiderlab.com with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/greatscottgadgets.schneiderlab.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

[grvvy]

$ journalctl -u snap.certbot.renew.timer
-- Logs begin at Fri 2026-01-16 20:45:01 UTC, end at Tue 2026-02-03 21:12:13 UTC. --
Feb 03 20:36:37 ip-x-x-x-x systemd[1]: snap.certbot.renew.timer: Succeeded.
Feb 03 20:36:37 ip-x-x-x-x systemd[1]: Stopped Timer renew for snap application certbot.renew.
Feb 03 20:36:38 ip-x-x-x-x systemd[1]: Started Timer renew for snap application certbot.renew.

[grvvy]

Stopping squid before running the dry run test renewal does in fact make it so that the dry run succeeds. Now to see if I can find a log of the timer event and what went wrong there to see if it's the same issue.

[grvvy]

I believe I found an entry in the certbot log that was triggered by the timer that is also failing with the same port 80 problem outlined above.

Is the proper course of action here to add a command to the timer that also stops and starts squid when trying to renew the cert? This doesn't seem ideal as it breaks the web server connection briefly. Is there a way to renew the cert and have the site never go down?

[grvvy]

I've manually renewed the certificate since I needed things up and running again, so if you click the link and it all looks normal, that's why.

[petercooperjr]

@grvvy

It's a bit complicated of a setup that someone else configured and documented.

Sounds like it might not have been documented quite enough.

I'll see if I can point you in the right direction, but other people may know more than I do about your setup.

@grvvy

I'm not sure how to find (if there exists) a log of previous attempts to renew the cert

As you may have already discovered, certbot stores its log in /var/log/letsencrypt/letsencrypt.log

@grvvy

I'm also unsure how the EC2 instance is running such a new version of certbot when I haven't run any updates on the system myself in a while.

It looks like certbot has been installed using snap. That's the method that the certbot maintainers recommend, and it keeps certbot up-to-date automatically by itself.

@grvvy

Manually testing automatic renewal shows a failure to renew due to another process using the same port (although the url it complains about is not the one I use to access the site, but it is one included in the configuration from the previous person). I'm guessing that if I stop squid and retry it then it might work, and perhaps that's what's causing issues with the renewal but I could use some more educated opinions before trying that.

Well, that's where it might help to understand why the system was set up the way it was. It sounds like your system may have certbot configured in "standalone" mode, where it runs its own web server for proving control over the domain name, but won't work if there's some other web server running at the same time.

Generally, you would have certbot integrate with the existing web server. There are plugins available for common web servers like Apache and Nginx, and you can use the "webroot" plugin if it's possible to just put a file somewhere on your server that your web application will serve. I'm not really familiar with squid proxy, so I'm not sure what the best way to integrate with it is.

@grvvy

Is the proper course of action here to add a command to the timer that also stops and starts squid when trying to renew the cert?

I wouldn't suggest putting it in the timer directly, since then it'd restart your web server multiple times a day. You can use the pre-hook and post-hook arguments to certbot to have it run a command that's needed in order to renew, such as shutting your web server down and starting it back up, something along the lines of

certbot reconfigure --cert-name greatscottgadgets.schneiderlab.com --pre-hook /root/shut-down-server.sh --post-hook /root/start-server-back-up.sh

You can get the actual cert name from certbot certificates, if my guess for it there based on your log files isn't right. And you'd need to create those scripts or directly include whatever commands need to be run for the hook.

@grvvy

This doesn't seem ideal as it breaks the web server connection briefly. Is there a way to renew the cert and have the site never go down?

I would agree that it's not ideal. I don't know why someone would have set it up that way. A lot might depend on why you're using squid to begin with, and if it has some way to integrate with the webroot plugin or some other way, which maybe someone else here has experience with.

[MikeMcQ]

I don't have anything to add to @petercooperjr good advice.

I just wanted to note that your server (squid? Jetty?) uses an incorrect cert file. It should be using the fullchain.pem created by Certbot but is using cert.pem instead. This sends out just your leaf cert and not the related intermediate certs. This can cause validation failures in browsers.

It is possible that has been wrong all along. You mentioned updating it 'manually' so perhaps something went wrong just recently.

See: SSL Checker

Or, try your own openssl command like this:

echo | openssl s_client -connect greatscottgadgets.schneiderlab.com:443 | head -20 

You should see an "unable to get local issuer" error. That means openssl could not "chase your cert chain" from that leaf to a trusted root on its system

Ignoring that cert error I saw these response headers. I assume Squid is probably terminating the inbound TLS (HTTPS) connection but I don't know much about Jenkins' products.

curl -Ik https://greatscottgadgets.schneiderlab.com
HTTP/1.1 200 OK
Server: Jetty(12.1.5)
X-Hudson: 1.395
X-Jenkins: 2.547
Via: 1.1 ip-10-0-1-164 (squid/4.10)

(note many headers omitted for brevity)

[grvvy]

Thanks for the response! Trying to wrap my head around all of this. Where can I see that it's using cert.pem? I'm not seeing it on SSL Checker or in the curl command output.

[grvvy]

I do see both the fullchain.pem and cert.pem under /etc/letsencrypt/live/greatscottgadgets.schneiderlab.com/ but I'm unsure how to get it to start using fullchain instead of cert here.

[MikeMcQ]

@grvvy

Where can I see that it's using cert.pem?

I don't know exactly. I am not expert with your servers. But, somewhere you must configure it to use a certificate.

@grvvy

I'm not seeing it on SSL Checker or in the curl command output.

The openssl command and that SSL Checker both show a problem with the chain your server sends. The SSL Checker says specifically it has no intermediates (below). The openssl command displays all the certs in the chain yet for yours it only shows a leaf and no intermediates. I know what to expect from that command from experience.

I also know the cert.pem from Certbot contains only the leaf and no intermediates. And, I know that fullchain.pem has the leaf and the needed intermediates. That's why I said that was likely what was wrong.

Note the above curl command without the -k will fail because of this. Other clients may have similar problems

curl -I https://greatscottgadgets.schneiderlab.com

[grvvy]

Gotcha. Thanks so much for explaining all of that! I'll do some digging and report back.

[grvvy]

For reference, here's a diagram of the entire setup that was provided by the person who set this up. Maybe this would provide some insight as to why things are configured the way they are. Certbot/LetsEncrypt living on the squid proxy server here.

[orangepizza]

for some weird reason it uses squid as reverse proxy. look for mention of cert.pem in /etc/squid/squid.conf
My first reaction was tear everything on EC2 side down and just hide jenkins behind a CDN like cloudflare though

[grvvy]

There's a line, that looks correct to my uneducated glance, mentioning the https port in squid.conf https_port 443 accel tls-cert=/etc/letsencrypt/live/greatscottgadgets.schneiderlab.com/fullchain.pem tls-key=/etc/letsencrypt/live/greatscottgadgets.schneiderlab.com/privkey.pem

[orangepizza]

squid : http_port configuration directive

   tls-cert=	Path to file containing an X.509 certificate (PEM format)
		to be used in the TLS handshake ServerHello.

		If this certificate is constrained by KeyUsage TLS
		feature it must allow HTTP server usage, along with
		any additional restrictions imposed by your choice
		of options= settings.

		When OpenSSL is used this file may also contain a
		chain of intermediate CA certificates to send in the
		TLS handshake.

		When GnuTLS is used this option (and any paired
		tls-key= option) may be repeated to load multiple
		certificates for different domains.

		Also, when generate-host-certificates=on is configured
		the first tls-cert= option must be a CA certificate
		capable of signing the automatically generated
		certificates.

it sounds you set it currectly: but it wording in manual sounds like if you are using GNUTLS as backend of squid it may not support chained certificate in this file

[grvvy]

Oh interesting. So “repeated to load multiple certificates for different domains” isn't what’s supposed to happen and is messing up the full chain? Sorry I’m not very well versed in this.

[orangepizza]

not that it's doing something wrong with GnuTLS but it only mentions doing certificate chaining when using Openssl is used.

When OpenSSL is used this file may also contain a
chain of intermediate CA certificates to send in the
TLS handshake.

[grvvy]

Gotcha. I don't see GnuTLS or OpenSSL mentioned anywhere in the documentation that was provided. Could it be assumed that GnuTLS is being used by default because it's a linux machine?

[orangepizza]

because squid-openssl is different package and apt list squid shows:

Package: squid
Version: 6.13-0ubuntu0.24.04.3
Priority: optional
Section: web
Origin: Ubuntu
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Original-Maintainer: Luigi Gangitano luigi@debian.org
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 8,321 kB
Pre-Depends: init-system-helpers (>= 1.54~), adduser
Depends: libc6 (>= 2.38), libcap2 (>= 1:2.10), libcom-err2 (>= 1.43.9), libcrypt1 (>= 1:4.1.0), libecap3 (>= 1.0.1), libexpat1 (>= 2.0.1), libgcc-s1 (>= 3.4), libgnutls30t64 (>= 3.8.1), libgssapi-krb5-2 (>= 1.17), libkrb5-3 (>= 1.10+dfsg~), libldap2 (>= 2.6.2), libltdl7 (>= 2.4.7), libnetfilter-conntrack3 (>= 1.0.1), libnettle8t64, libpam0g (>= 0.99.7.1), libsasl2-2 (>= 2.1.28+dfsg1), libstdc++6 (>= 13.1), libsystemd0, libtdb1 (>= 1.2.7+git20101214), libxml2 (>= 2.7.4), netbase, logrotate (>= 3.5.4-1), squid-common (>= 6.13-0ubuntu0.24.04.3), lsb-base, libdbi-perl, ssl-cert
Recommends: libcap2-bin, ca-certificates
Suggests: squidclient, squid-cgi, squid-purge, resolvconf (>= 0.40), smbclient, ufw, winbind, apparmor
Conflicts: squid-openssl
Homepage: http://www.squid-cache.org
Download-Size: 2,743 kB
APT-Sources: Index of /ubuntu/ noble-updates/main amd64 Packages
Description: Full featured Web Proxy cache (HTTP proxy GnuTLS flavour)
Squid is a high-performance proxy caching server for web clients, supporting
FTP, ICY and HTTP data objects.

remove squid package and install squid-openssl instead should work: not sure doing so will delete config file too