Certs stopped updating, pls help


#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: dom.etogo.net.

I ran this command: sudo certbot

It produced this output: Failed authorization procedure. dom.etogo.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://dom.etogo.net/.well-known/acme-challenge/h7WjzOaFXsuqz8ETqLT9YNYaNG5_7NfJRm6Bd6xnxQw: “\n\n403 Forbidden\n\n

Forbidden

\n<p”, phone.etogo.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://phone.etogo.net/.well-known/acme-challenge/PWcIQWexMDBd1yccrXvjYwB-qlblcJc3SpOlNFJP0QM: “\n\n403 Forbidden\n\n

Forbidden

\n<p”, billing.design (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://billing.design/.well-known/acme-challenge/FiDj3k35Eql2zvhQlIrDim7I1uZFD4ekP7_POdHfbjQ: "\n\n\n\n\n\n\n\n\n\n<!doctype html>\n\n\n \n\n <meta charset=“UTF-8”>\n <meta name=“viewport” content=“width=device”, opinions.expert (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://opinions.expert/.well-known/acme-challenge/RFD4b5Z2_erLbEC9Hpdzb899yvFNdYN7DV6gyHAiLu4: “\n404 Not Found\n\n

404 Not Found

\n/.well-known/acme-challenge/RFD4b5Z2_erLbEC9Hpdzb”

IMPORTANT NOTES:

My web server is (include version): Apache/2.4.25 (Debian)

The operating system my web server runs on is (include version): Debian 9.6

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no


#2

Your current active cert expires in 1 month and 2 days.
So, since you are not pressed for time, I will not rush into a quick-fix or temporary-fix…

We need to understand how the cert was issued and if renewed.
To help with that, you can show the renewal.conf file located at /etc/letsencrypt/renewal/{your.site}.conf

The error shown is 403 forbidden - have you made any changes recently (last 60 days) that might have affected access to the challenge folder?


#3

Sure.
I actually have several/multiple domains on 2 hosts. one of them was multihomed and I removed one interface 25th of dec or something. Today I received a warning email that some of my domains will expire in 9 days, I trtied to run certbot and found that. Double checked dns and other stuff and found nothing wrong (although there is something, for sure)
I tried to create the folder by hands, created a text file there and checked it with apache/webbrowser - everything was working fine.
The logging is so poor, it’s really hard to understand that’s wrong. Here’s the file (billing.design is just another name here, dunno is it safe to expose account btw?):

renew_before_expiry = 30 days

version = 0.25.0
archive_dir = /etc/letsencrypt/archive/billing.design
cert = /etc/letsencrypt/live/billing.design/cert.pem
privkey = /etc/letsencrypt/live/billing.design/privkey.pem
chain = /etc/letsencrypt/live/billing.design/chain.pem
fullchain = /etc/letsencrypt/live/billing.design/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = apache
installer = apache
account = b5d0fc2058f6cca6c2455251e8281e46

To be more specific - I do have current debian version with all updates installed, running latest apache available in deb repos, and Ive been using certbot for years with (almost))) no problems. And now I have no idea how to fix that.


#4

Could you give an example of a link with a sample file that we could check?

(The authenticator = apache means that Certbot doesn’t actually use that path, unlike authenticator = webroot; if you’re sure this works you could also choose change the authenticator over to webroot and tell Certbot what webroot path to use. However, it would still be interesting to see a sample file to rule out the possibility of some other kinds of errors.)


#5

Hi @Viktr

there is one certificate with 10 domain names:

CN=billing.design
	08.11.2018
	06.02.2019
	base.etogo.net, billing.design, console.etogo.net, dom.etogo.net, 
intelligent-systems.com.ua, net.etogo.net, opinions.expert, phone.etogo.net, 
prosto.etogo.net, webmail.etogo.net - 10 entries

Looks like you have created another certificate earlier with a different set of domain names. But you have this valide certificate, so you can ignore the mail.


#6

Yes, I do have more than 10 names here on 2 physical hosts. But my question is, why did I get this error I mentioned above? I’ve got it on both hosts, it’s strange. I suppose, letsencrypt package created 2 different accounts on both hosts, but the error is the same. This is the problem, not the certs. I suppose it won’t gone itself, so i’m trying to understand what’s wrong


#7

I’m not sure I understand how it works in details, I supposed it uses modrewrite or modproxy or something alike to redirect actual request to the letsencrypt site in order to check am I authorized to manage this domain. I saw an error in the apache logs saying client denied by server configuration: /var/lib/letsencrypt/http_challenges/VevE99g9kjX4V1xa4VCH2ffpLXlAFAoSLnKD09Pom-A so I created http://dom.etogo.net/.well-known/acme-challenge/test.txt to test.


#8

Yes, that’s correct. If you could share your Apache log and Apache configuration with us, it might help us understand more about what went wrong. In the meantime, you could probably also make the renewal work with -a webroot -w [the_directory_that_contains_your_.well-known_directory]. (The webroot authenticator literally does create files in an existing directory, while the apache authenticator does redirect the request to a custom directory under /var/lib/letsencrypt/http_challenges.)


#9

Is there any information about why it would be denied? Certbot tries to avoid that kind of thing. What kind of client-denying stuff is in the Apache configuration?


#10

And what was the result of that test?
Where did you test from?
From where I’m sitting this test failed.


#11

It’s a pretty standard config, dunno how to share, nothing special and, I suppose more important, nothing was changed for last months except this nic removal. The apache uses its config for years, nothing changed. Here’s the log: http://dom.etogo.net/.well-known/acme-challenge/logs.tgz


#12

I see your attempt in the logs. It failed because you tried to access simultaneously with me - I just ran certbot again to get fresh error log. I am testing using torBrowser and the result is ok - everything is accessible.


#13

I don’t see what you see.
What I see says I don’t have any access to port 80.

–2019-01-04 01:23:40-- (try: 2) http://dom.etogo.net/.well-known/acme-challenge/test.txt
Connecting to dom.etogo.net (dom.etogo.net)|46.164.150.51|:80… failed: Connection timed out.
Retrying.


#14

You might have seen my test rather than @rg305’s test. To me, this loads properly!

Edit: oddly I see dom.etogo.net resolving to 217.147.174.254, not 46.154.150.51.


#15

Looks like there’s something wrong with the dns, dunno on my or your side:

`dig dom.etogo.net @8.8.8.8

; <<>> DiG 9.10.3-P4-Debian <<>> dom.etogo.net @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30350
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;dom.etogo.net. IN A

;; ANSWER SECTION:
dom.etogo.net. 9297 IN A 217.147.174.254

;; Query time: 33 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jan 04 03:57:14 EET 2019
;; MSG SIZE rcvd: 58`

Could you please share your resolver configuration?


#16

@Viktr, it would also still be really helpful to see your Apache configuration because I think there’s probably a Certbot bug here in the way that Certbot failed to tell Apache to allow the files to be downloaded from within /var/lib/letsencrypt.


#17

I am new here, please help me - how could I send it directly to you?
Ty


#18

well, I suppose nothing secret there http://dom.etogo.net/.well-known/acme-challenge/apache.etc.tar.gz pls let me know so I’d delete it.


#19

Thanks, I’ve downloaded it.

One thing I notice is that the site that’s failing with --apache has

        <Directory />
                Order deny,allow
                deny from all

                AllowOverride None
        </Directory>

With only one exception, your other virtualhosts don’t have this same policy. I think that accounts for the difference in whether they succeed or fail.

(However, this is still a Certbot bug because Certbot should be able to deal correctly with this situation!)


#20

Yes, I see :wink:
Well, this is for the root dir, don’t know is it really necessary butI assume you’re correct. And the most strange thing is, that I changed nothing and it was working - how could it be?