403 forbidden certificate creation error

Help
1

Sorry for my bad english language, I'm trying my best

My domain is: coffebuisness.shop

Tell me if you need additional information

I'm using structure that includes:

  • Proxy server using haproxy (domain A-record is binded on him)
  • Main server with docker containers nginxproxy/acme-companion and nginxproxy/nginx-proxy
  • php instances as websites (php_instance:latest)

My web server is : nginx 1.27.3 (docker container)

The operating system my web server runs on is : Ubuntu 24.04

My hosting provider, if applicable, is: Cloudzy

I can login to a root shell on my machine : yes

I'm using a control panel to manage my site : no

The main problem is this error:

[Thu Jan 23 17:16:22 UTC 2025] Pending. The CA is processing your order, please wait. (1/30)
[Thu Jan 23 17:16:25 UTC 2025] coffebuisness.shop: Invalid status. Verification error details: 89.47.249.4: Invalid response from http://coffebuisness.shop/.well-known/acme-challenge/NEKY_n61AtgGQCYCGYfaZcDaEf14dZs3LE2bIut8CeU: 403
[Thu Jan 23 17:16:25 UTC 2025] Please check log file for more details: /dev/null

I was trying different domains, output is the same
If I go directly to a link, it shows 403 Forbidden error
Permissions on files and folder are 0755, owner and group is root

This is brand-new test VPS, I am mentions that because I have my production server, that is working on Ubuntu 22.04, same configuration, and no error

Trying to handle this error for 2 days, no chanses, please help me :innocent:

2

My best guess is you have a Palo Alto brand firewall blocking access to the ACME HTTP Challenge requests.

See below the first URI gets redirected to HTTPS just like "home" page requests. But, the second adds a slash and gets a 403. We have seen this pattern before with Palo Alto firewalls.

If you have one of those check the acme-protocol setting in the "application" section

image
image1461×663 138 KB

1 Like
3

It might be my solution, I'll ask VPS provider about this Palo Alto firewall and tell you later, because I have no access to this "application"
Thanks in advance!

1 Like
4

VPS provider answered me that they are not using any firewall, and they tried to renew and issue certificates without any issue, can you help after that information ?

1 Like
5

Then it must be something on your system sending 403 Forbidden.

The response headers say "Server: nginx". So, check your nginx server access and/or error logs to see if it is rejecting these.

You may need to check HAProxy logs as well.

Use the same curl example I showed or this test site: Let's Debug

2 Likes
6

I can't find an issue, every config check says that everything is okay, and on my other server, I have totally the same template and everything is working just fine, for haproxy, and for nginx as well
If I provide the configs, maybe you can see that I can't ?

7

Maybe but that is a lot of components (hosting / ISP, HAProxy, nginx, NPM, containers) to review. The Let's Debug test now shows a 503 Service Temporarily Unavailable. So, something changed because it was getting a 403 Forbidden earlier.

Do the below curl requests work for you? Make sure to try from the public internet and not just your local network.

You should ensure these curl requests reach the system that is requesting the certificate. What program did you use? And, what was the exact command? Because usually with NPM systems we see Certbot but that log looks like it might have been something else.

In the curl tests below usually the "server" header says "LiteSpeed" when using NPM. That it says "nginx" for some requests is interesting. And, it does not have any server: header for other http requests.

# "home" page request - note redirect and no server: header
curl -i http://coffebuisness.shop
HTTP/1.1 302 Found
content-length: 0
location: https://coffebuisness.shop/
cache-control: no-cache

# A partial HTTP Challenge request - same redirect
curl -i http://coffebuisness.shop/.well-known/acme-challenge
HTTP/1.1 302 Found
content-length: 0
location: https://coffebuisness.shop/.well-known/acme-challenge
cache-control: no-cache

# Adding the slash or sample token gets failure - not a redirect
# And see the server:nginx header
curl -i http://coffebuisness.shop/.well-known/acme-challenge/
curl -i http://coffebuisness.shop/.well-known/acme-challenge/Test404
HTTP/1.1 503 Service Temporarily Unavailable
server: nginx
date: Sun, 26 Jan 2025 16:58:13 GMT
content-type: text/html
content-length: 190

# The certificate used for the HTTPS connection is this.  
# Do you know what this is?
openssl s_client -connect coffebuisness.shop:443

subject=CN = v2sstest.node.lan
issuer=CN = v2sstest.node.lan
notBefore=Jan 24 20:08:49 2025 GMT
notAfter=Dec 31 20:08:49 2124 GMT
1 Like
8

No, I just tried another domain, now I've restored the one in the topic, sorry

9

And I've created file Test404, with "Test404" text inside, still 403 Forbidden ...

10

Presently I see

For the ACME Challenge check to test for Palo Alto Firewall (the firewall presently doesn't seem might or might not to be an issue).

$ curl -Ii http://coffebuisness.shop/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
HTTP/1.1 403 Forbidden
server: nginx
date: Sun, 26 Jan 2025 18:20:50 GMT
content-type: text/html
content-length: 146
vary: Accept-Encoding
vary: Accept-Encoding

And just the quick and simple ACME challenge check.

$ curl -Ii http://coffebuisness.shop/.well-known/acme-challenge/sometestfile
HTTP/1.1 403 Forbidden
server: nginx
date: Sun, 26 Jan 2025 18:20:52 GMT
content-type: text/html
content-length: 146
vary: Accept-Encoding
vary: Accept-Encoding

Since I see server: nginx,
Please show the output of this command sudo nginx -T that is a capital T

11

Please also show the output of sudo ls -la for the directory you placed the file Test404 on the server.

1 Like
12

(post deleted by author)

13

(post deleted by author)

14

Looks like you have your acme-companion in NPM using acme.sh. And, you've placed a Test404 file there too.

Now you have to figure out why the HTTP request from the public internet cannot find it.

You probably need to consult with NPM support or even consult with acme-companion support. You have a complex mix of server products and HTTP flows. We are not a general networking or server config help forum.

We've given you tools to help debug this. I don't think you'll find anyone else here to help further. You need to find a set of products that you can manage. You might consider simpler options to achieve your goal.

I saw your nginx port 80 proxies to another service. Review your configuration of that too:
proxy_pass http://coffebuisness.shop-hashexample/

Best of luck

3 Likes