Ubuntu AWS SSH trouble renewing certificates

Help
1

My domain is: wardofcode.com, match.wardofcode.com, wickedsales.wardofcode.com

I ran this command: sudo certbot renew

It produced this output:

Processing /etc/letsencrypt/renewal/match.wardofcode.com.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Attempting to renew cert (match.wardofcode.com) from /etc/letsencrypt/renewal/match.wardofcode.com.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8dba87cc50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)). Skipping.

Processing /etc/letsencrypt/renewal/wardofcode.com.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Attempting to renew cert (wardofcode.com) from /etc/letsencrypt/renewal/wardofcode.com.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8dba723438>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)). Skipping.

Processing /etc/letsencrypt/renewal/wickedsales.wardofcode.com.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Attempting to renew cert (wickedsales.wardofcode.com) from /etc/letsencrypt/renewal/wickedsales.wardofcode.com.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8dbb03d780>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)). Skipping.

I'm also having the same type of error on my other domains as well from the renew command

Super new to all this so not super sure about the following three

My web server is (include version): Ubuntu 18.04.3 LTS (GNU/Linux 5.4.0-1048-aws x86_64)

The operating system my web server runs on is (include version): Ubuntu 18.04.3 LTS (GNU/Linux 5.4.0-1048-aws x86_64)

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): not sure

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.31.0

I'm using gitbash to ssh into the AWS ubuntu server. I originally was able to get certificates for my domain a few years back no problem, and auto renew has occurred without me needing to do anything previously. But now renewal won't occur.

sudo certbot certificates

produces the following (note, I'm excluding the failed checks for other domains for legibility):

Saving debug log to /var/log/letsencrypt/letsencrypt.log
OCSP check failed for /etc/letsencrypt/live/match.wardofcode.com/cert.pem (are we offline?)
OCSP check failed for /etc/letsencrypt/live/wardofcode.com/cert.pem (are we offline?)
OCSP check failed for /etc/letsencrypt/live/wickedsales.wardofcode.com/cert.pem (are we offline?)

Found the following certs:
Certificate Name: match.wardofcode.com
Domains: match.wardofcode.com
Expiry Date: 2022-04-28 23:29:51+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/match.wardofcode.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/match.wardofcode.com/privkey.pem
Certificate Name: wardofcode.com
Domains: wardofcode.com
Expiry Date: 2022-06-11 03:21:20+00:00 (VALID: 8 days)
Certificate Path: /etc/letsencrypt/live/wardofcode.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/wardofcode.com/privkey.pem
Certificate Name: wickedsales.wardofcode.com
Domains: wickedsales.wardofcode.com
Expiry Date: 2022-04-28 23:30:11+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/wickedsales.wardofcode.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/wickedsales.wardofcode.com/privkey.pem

I've been looking at a lot of other posts for similar issues for weeks now and I'm pretty confused what I need to do to fix this.

2

hmm...

Please show both outputs:
dig +short acme-v02.api.letsencrypt.org
dig +short acme-v02.api.letsencrypt.org @8.8.8.8

7 Likes
3

Thank you for the quick response!

code:
dig +short acme-v02.api.letsencrypt.org

response:

;; connection timed out; no servers could be reached

code:
dig +short acme-v02.api.letsencrypt.org @8.8.8.8

response:

prod.api.letsencrypt.org.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
172.65.32.248

1 Like
4

Well, it seems that your default DNS servers are unable to resolve your DNS requests.

Please show:
cat /etc/resolv.conf

5 Likes
5

Code:
cat /etc/resolv.conf

Response:

cat: /etc/resolv.conf: No such file or directory

But it does show that "/etc/resolv.conf" is inside "/etc/" when I do

ls /etc/

Not sure if this helps but

ls -al /etc/resolv.conf

produces

lrwxrwxrwx 1 root root 39 Oct 2 2019 resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

6

What shows?:
cat /run/systemd/resolve/stud-resolv.conf TYPO!

4 Likes
7

cat: /run/systemd/resolve/stud-resolv.conf: No such file or directory

This time I cant actually find a /run/systemd/resolve directory either

9

Try:
locate stub-resolv.conf
find / -name stub-resolv.conf

What shows?:
cat /run/systemd/resolve/stub-resolv.conf

4 Likes
10

Nothing happens

A very long list (1052 lines) of directories such as:

find: ‘/lost+found’: Permission denied
find: ‘/proc/tty/driver’: Permission denied
find: ‘/proc/1/task/1/fd’: Permission denied
find: ‘/proc/1/task/1/fdinfo’: Permission denied
find: ‘/proc/1/task/1/ns’: Permission denied
find: ‘/proc/1/fd’: Permission denied
....
find: ‘/proc/402/map_files’: Permission denied
find: ‘/proc/402/fdinfo’: Permission denied
find: ‘/proc/402/ns’: Permission denied
find: ‘/proc/408/task/408/fd’: Permission denied
find: ‘/proc/408/task/408/fdinfo’: Permission denied
find: ‘/proc/408/task/408/ns’: Permission denied
...
find: ‘/var/lib/snapd/cookie’: Permission denied
find: ‘/var/lib/snapd/cache’: Permission denied
find: ‘/var/lib/snapd/void’: Permission denied
...
find: ‘/etc/letsencrypt/accounts’: Permission denied
find: ‘/etc/letsencrypt/live’: Permission denied
find: ‘/etc/letsencrypt/archive’: Permission denied
find: ‘/etc/letsencrypt/keys’: Permission denied
find: ‘/etc/sudoers.d’: Permission denied
find: ‘/etc/polkit-1/localauthority’: Permission denied
find: ‘/etc/ssl/private’: Permission denied
find: ‘/run/lxcfs’: Permission denied
find: ‘/run/sudo’: Permission denied
find: ‘/run/postgresql/10-main.pg_stat_tmp’: Permission denied
find: ‘/run/cryptsetup’: Permission denied
find: ‘/run/lvm’: Permission denied
find: ‘/run/systemd/ask-password-block’: Permission denied
find: ‘/run/systemd/unit-root’: Permission denied
find: ‘/run/systemd/inaccessible’: Permission denied
find: ‘/run/lock/lvm’: Permission denied

The "..." denotes multiple lines skipped. They all say Permission denied. Do you want the full list of 1052 lines?

I also tried sudo find / -name stub-resolv.conf and that processed for a while just like without sudo but then nothing.

Same as before:

ubuntu@ip-***-**-**-**:~$ cat /run/systemd/resolve/stub-resolv.conf
cat: /run/systemd/resolve/stub-resolv.conf: No such file or directory
ubuntu@ip-***-**-**-**:~$ ls -l /run/systemd/resolve/stub-resolv.conf
ls: cannot access '/run/systemd/resolve/stub-resolv.conf': No such file or directory
ubuntu@ip-***-**-**-**:~$ ls -l /run/systemd/resolve/
ls: cannot access '/run/systemd/resolve/': No such file or directory
ubuntu@ip-***-**-**-**:~$ ls -l /run/systemd/
total 0
drwxr-xr-x 2 root root 40 Apr 1 16:16 ask-password
drwx------ 2 root root 60 May 12 20:15 ask-password-block
drwxr-xr-x 7 root root 180 Apr 1 16:16 generator
drwxr-xr-x 3 root root 60 Apr 1 16:16 generator.early
drwxr-xr-x 4 root root 140 Apr 1 16:16 generator.late
d--------- 3 root root 160 Apr 1 16:16 inaccessible
drwxr-xr-x 2 root root 80 Apr 1 16:16 inhibit
drwxr-xr-x 2 root root 60 Apr 1 16:16 initctl
drwxr-xr-x 3 root root 180 Apr 1 16:16 journal
drwxr-xr-x 2 root root 40 Apr 1 16:16 machines
drwxr-xr-x 5 systemd-network systemd-network 120 Jun 6 20:27 netif
drwxr-xr-x 2 root root 80 Apr 1 16:16 network
srwxrwxrwx 1 root root 0 Apr 1 16:16 notify
srwxrwxrwx 1 root root 0 Apr 1 16:16 private
drwxr-xr-x 2 root root 60 Apr 1 16:16 seats
drwxr-xr-x 2 root root 80 Jun 6 20:35 sessions
-rw-r--r-- 1 root root 0 Apr 1 16:16 show-status
drwxr-xr-x 2 root root 40 Apr 1 16:16 shutdown
drwxr-xr-x 2 root root 40 Apr 1 16:16 system
drwxr-xr-x 2 root root 80 Jun 6 20:35 transient
drwx------ 2 root root 40 Apr 1 16:16 unit-root
drwxr-xr-x 2 root root 1280 Jun 6 20:35 units
drwxr-xr-x 2 root root 60 Jun 6 20:35 users

11

I don't have experience with Ubuntu nor AWS, but a quick Google search leads me to a guide which use "netplan" to set the DNS servers in Ubuntu. This seems to be the default behaviour/method? According to that guide anyway. You should probably check it out:

4 Likes
12

Thanks @Osiris for the resource. I was able to read through, but I wasn't able to get very far in applying the steps they showed.

The article mentions

Netplan configuration files are stored in the /etc/netplan directory. You’ll probably find one or two YAML files in this directory. The file name may differ from setup to setup. Usually, the file is named either 01-netcfg.yaml or 50-cloud-init.yaml but in your system, it may be different.

So in my aws server I have '50-cloud-init.yaml' in the '/etc/netplan' directory.

The article suggests:

The file’s contents will look something like the following:

/etc/netplan/01-netcfg.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      dhcp4: no
      addresses:
        - 192.168.121.199/24
      gateway4: 192.168.121.1
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]

When I do:

sudo nano /etc/netplan/50-cloud-init.yaml

shows an empty file with 0 lines.

I also checked out a line the article mentioned to verify the DNS server:

systemd-resolve --status | grep 'DNS Servers' -A2

And it produced:

     DNS Servers: ***.**.*.*
      DNS Domain: us-east-2.compute.internal

I hid the IPv4 address that was listed just in case as it looks very similar to my private IP address from AWS but not quite the same. But the DNS Domain is at the end of my full Private IP DNS name from AWS Dashboard.

So I don't know if that helps at all or not, but it's new information. Also, the DNS Server font-color in gitbash is red.

That's all I got so far

1 Like
13

As I said, I don't have experience with Ubuntu (nor systemd for that matter, Gentoo uses OpenRC by default), but is that DNS server you've redacted actually working as such?

4 Likes
14

Hey, yeah I appreciate you going out on a limb to help. The Linux waters are still a little murky to me too, let alone Ubuntu.

A lot of IP and infrastructure stuff is still a little fuzzy to me so I just redacted it, but I don't think its anything sensitive after looking more into it. I think it's affiliated with Amazon.

DNS Servers
I uploaded screenshot to show the red. I don't know why it's red.

What's so weird is I haven't had any issues before now. I did do some deleting of older test sites to free up space in the AWS storage and I didn't do any certificate removal from those old domains. But I don't know if that has any connection or why it would stop things from connecting.

EDIT: The sites are working now! I have no idea what happened, but all the sites are renewed now! So I'm guess maybe there was some internal issue with the server or maybe something else I'm not aware of, but I was just doing a dry run and test cert and both were telling me everything was all good, so I checked the sites again and BAM! And these sites weren't renewing for over a month before this point. :thinking:

So thank you for all your combined help! I've learned a lot. I just have no idea why the issue got fixed :sweat_smile: :rofl:

1 Like
16

DNS failed to resolve the required name = renewals failed.
Renewals now work = DNS must be working now.

Why DNS failed? Why DNS works now?
Only AWS would know.

What I do know is that I wouldn't rely on such a fragile DNS configuration.
One single point of failure [that you don't know anything about - who should you even call when it fails?]
So.....
My advice is to change your single DNS entry to multiple entries [from DNS systems that are well known and easy to deal with]

7 Likes
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.