Error 429 and Operation not Permited

[giladsky]

Hello And thank you for taking the time to read
I have a domain giladsky.com and a subdomain chat.giladsky.com
everything was working fine, i have a weekly cronjob to renew certificates, yesterday on my subdomain i received an error 429, if i understood correct i did not hit the rate limit
this is from my log file

Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}

and also i started to see this error

chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/uTuEg4xH1SJUlb_0h_UN9zvmoFGDQTzNN6IBXEXFc1I':>
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known': Operation not permitted

even though when checking permission all is set like before

Thank you for taking the time if you can please help me

Thank you
Gilad

[sahsanu]

Hello @giladsky,

Just to be clear, the problem is renewing chat.giladsky.com right?

@giladsky

i received an error 429, if i understood correct i did not hit the rate limit

You are hitting the rate limit of 5 failures per account, per hostname, per hour.

@giladsky

i have a weekly cronjob to renew certificates

You didn't say it but I suppose you are using acme.sh. What is the user where you have defined that cron job and what is the command you are executing?

Could you please show the output of this command?

namei -mo /var/www/letsencrypt/.well-known/acme-challenge/

Is /var/www/letsencrypt/.well-known/acme-challenge/ mounted in a normal partition, I mean it is not a samba share or nfs, etc.?

Show also the output of these commands:

df -h /var/www/letsencrypt/.well-known/acme-challenge/
df -ki /var/www/letsencrypt/.well-known/acme-challenge/

Cheers,
sahsanu

[giladsky]

Thank you @sahsanu for editing the post and your prompt responsd

I am a newbie

@sahsanu

You are hitting the rate limit of 5 failures per account, per hostname, per hour.

how i hit the limit if the cronjob runs only once a week?

@sahsanu

You didn't say it but I suppose you are using acme.sh. What is the user where you have defined that cron job and what is the command you are executing?

i defined the the user acmeuser in the group of www-data
i do use acme.sh and the command is

sudo -u acmeuser "/home/acmeuser/.acme.sh"/acme.sh --cron --home "/home/acmeuser/.acme.sh"

@sahsanu

namei -mo /var/www/letsencrypt/.well-known/acme-challenge/

the output is

 drwxr-xr-x root     root     /
 drwxr-xr-x root     root     var
 drwxr-x--- www-data www-data www
 drwxrwxr-x www-data www-data letsencrypt
 drwxrwxr-x www-data www-data .well-known
 drwxrwxr-x www-data www-data acme-challenge

@sahsanu

Is /var/www/letsencrypt/.well-known/acme-challenge/ mounted in a normal partition, I mean it is not a samba share or nfs, etc.?

it is mounted in a normal partition

@sahsanu

df -h /var/www/letsencrypt/.well-known/acme-challenge/

the output is

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       8.8G  4.9G  3.5G  59% /

@sahsanu

df -ki /var/www/letsencrypt/.well-known/acme-challenge/

the output is

Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/sda3      589824 140467 449357   24% /

Again Thank you very much!

[sahsanu]

@giladsky

how i hit the limit if the cronjob runs only once a week?

I don't know, maybe you have another cron job out there or you executed it manually.

I see no problem neither in your dirs nor your file system.

Please, show the output of this command:

ls -laR /var/www/letsencrypt/

[giladsky]

@sahsanu

I don't know, maybe you have another cron job out there or you executed it manually.

I have another crunjob running on giladsky.com also weekly but had never had issued and renewed the certificate on December 27
and i do not run in manually besides the cronjob

@sahsanu

ls -laR /var/www/letsencrypt/

the output is

total 12
drwxrwxr-x 3 www-data www-data 4096 Nov 15 04:18 .
drwxr-x--- 3 www-data www-data 4096 Nov 15 04:18 ..
drwxrwxr-x 3 www-data www-data 4096 Nov 15 04:18 .well-known

/var/www/letsencrypt/.well-known:
total 12
drwxrwxr-x 3 www-data www-data 4096 Nov 15 04:18 .
drwxrwxr-x 3 www-data www-data 4096 Nov 15 04:18 ..
drwxrwxr-x 2 www-data www-data 4096 Jan 17 13:27 acme-challenge

/var/www/letsencrypt/.well-known/acme-challenge:

total 8
drwxrwxr-x 2 www-data www-data 4096 Jan 17 13:27 .
drwxrwxr-x 3 www-data www-data 4096 Nov 15 04:18 ..

[sahsanu]

As far as I can see, the code in acme.sh that is triggering the chown problem is this:

if [ ! "$usingApache" ]; then
          if webroot_owner=$(_stat "$_currentRoot"); then
            _debug "Changing owner/group of .well-known to $webroot_owner"
            if ! _exec "chown -R \"$webroot_owner\" \"$_currentRoot/.well-known\""; then
              _debug "$(cat "$_EXEC_TEMP_ERR")"
              _exec_err >/dev/null 2>&1
            fi
          else
            _debug "not changing owner/group of webroot"
          fi
        fi

Could you please execute this?

sudo -u acmeuser "/home/acmeuser/.acme.sh"/acme.sh --home "/home/acmeuser/.acme.sh" --renew -d chat.giladsky.com --debug 2

Once executed, paste the output of previous command and the log that should be created here /home/acmeuser/.acme.sh/acme.sh.log

To paste the log you can use some site like https://paste.ubuntu.com/

I'm afraid I'll leave in a few minutes but some community buddy could help you or maybe @Neilpang (acme.sh developer) could take a look to this issue.

If I can't answer today I'll be back tomorrow

[giladsky]

Dear @sahsanu Thank you for taking the time

@sahsanu

paste the output of previous command

output of this command
https://paste.ubuntu.com/p/YhTGc26sPn/

@sahsanu

log that should be created here /home/acmeuser/.acme.sh/acme.sh.log

Log file
https://paste.ubuntu.com/p/DNQmSpt3Sk/

Again Thank you!
and besides do you know what can i do regarding the error 429?

[sahsanu]

@giladsky

and besides do you know what can i do regarding the error 429?

You can do nothing, just wait but you just need to wait 1 hour so that limit has gone already.

Regarding the problem, seems acme.sh is writing the challenge file as acmeuser or maybe as root (I tested it on my server and it executes the command as the user but it writes the files as root, I'm trying to know the reason) (forget what I say about files written as root, it was me doing a wrong redirection in current shell instead of spawning a shell) and then it tries to modify the owner to www-data but it can't (chown error) but it continues the process and your nginx server can't serve the challenge and gives a 403 forbidden error to Let's Encrypt so it can't validate the challenge to issue the cert for your domain.

Why is it happening now but it worked before? No idea, are you sure you didn't change anything?

Instead of launching the command as root and using sudo you could try to login as acmeuser and execute the command:

/home/acmeuser/.acme.sh/acme.sh --home "/home/acmeuser/.acme.sh" --renew -d chat.giladsky.com --debug 2

or with root but using su instead of sudo:

su - acmeuser -c "/home/acmeuser/.acme.sh/acme.sh --home "/home/acmeuser/.acme.sh" --renew -d chat.giladsky.com --debug 2"

Tomorrow more, too late here.

[sahsanu]

I forgot to say that if the above doesn't work you could also try to add www-data user to acmeuser group (just to test).

[giladsky]

Again @sahsanu Thank you

@sahsanu

are you sure you didn't change anything?

yes i did not change anything

After one hour, i am not getting no more error 429
but still the error of chown "operation not permitted" even when logged as acmeuser

@sahsanu

add www-data user to acmeuser group

they were all the time in the same group,
please if you think on anything else let me know

Thank you

[sahsanu]

I think you have been using the apache mode to issue your certificate. Do you remember the command used to issue the certificate the first time?

Please, show the ouput of these commands:

ls -laR /home/acmeuser/.acme.sh/

cat /home/acmeuser/.acme.sh/chat.giladsky.com/chat.giladsky.com.conf

[giladsky]

Hello @sahsanu

@sahsanu

Do you remember the command used to issue the certificate the first time?

acme.sh --issue -d chat.giladsky.com --keylength 4096 -w /var/www/letsencrypt --key-file /etc/letsencrypt/rsa-certs/privkey.pem --ca-file /etc/letsencrypt/rsa-certs/chain.pem --cert-file /etc/letsencrypt/rsa-certs/cert.pem --fullchain-file /etc/letsencrypt/rsa-certs/fullchain.pem

acme.sh --issue -d chat.giladsky.com --keylength ec-384 -w /var/www/letsencrypt --key-file /etc/letsencrypt/ecc-certs/privkey.pem --ca-file /etc/letsencrypt/ecc-certs/chain.pem --cert-file /etc/letsencrypt/ecc-certs/cert.pem --fullchain-file /etc/letsencrypt/ecc-certs/fullchain.pem

@sahsanu

ls -laR /home/acmeuser/.acme.sh/

https://paste.ubuntu.com/p/8y72KS6bcg/

@sahsanu

cat /home/acmeuser/.acme.sh/chat.giladsky.com/chat.giladsky.com.conf

https://paste.ubuntu.com/p/7nM3rF3vwQ/

Thank you

[sahsanu]

I've simulated your situation, same user, same perms, etc. and I get the same chown errors:

[Tue Jan 19 17:22:26 UTC 2021] chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known': Operation not permitted

But I get the certificate so seems the chown error is not preventing that you get the certificate so maybe there is problem in nginx conf.

Using your acmeuser write a test file:

echo "this is a test file" > /var/www/letsencrypt/.well-known/acme-challenge/test-acme

Once done let me know and/or test it yourself.

http://chat.giladsky.com/.well-known/acme-challenge/test-acme

[giladsky]

Thank you for taking your time to try to resolve this

@sahsanu

Using your acmeuser write a test file:

i created the test file as acmeuser after testing i receive 404 error
so maybe it worked? in my nginx conf it mentioned that if something not exist to assume it is a chat room
i left it so you can review it yourself at
http://chat.giladsky.com/.well-known/acme-challenge/test-acme

do you need me to includ my nginx conf ?

[sahsanu]

@giladsky

i created the test file as acmeuser after testing i receive 404 error
so maybe it worked? in my nginx conf it mentioned that if something not exist to assume it is a chat room

No, it didn't work but at least we know your nginx conf is not serving the http-01 challenge in the right dir.

@giladsky

do you need me to includ my nginx conf ?

Yes please, show your conf.

[giladsky]

@sahsanu

Yes please, show your conf.

https://paste.ubuntu.com/p/J922GgCMjK/

please review my nginx conf
and again thank you

[giladsky]

Dear @sahsanu
I wanted to tell you thank you very much

@sahsanu

No, it didn't work but at least we know your nginx conf is not serving the http-01 challenge in the right dir.

i made this changes in my nginx.conf to

root /var/www/letsencrypt

}
location = /.well-known/acme-challenge/ {
   return 404;

i removed the 404 section

and now everything is working well!

Again Thank you very much for taking the time!

[giladsky]

any way if you can review it please to see that is the right fix

Thank you

[giladsky]

@sahsanu

Using your acmeuser write a test file:

echo "this is a test file" > /var/www/letsencrypt/.well-known/acme-challenge/test-acme

Once done let me know and/or test it yourself.

http://chat.giladsky.com/.well-known/acme-challenge/test-acme

i am asking because even though i managed to get the new certificate going to this pass i still get 404
should i see the test file?

[sahsanu]

@giladsky

I wanted to tell you thank you very much

You are welcome

@giladsky

i am asking because even though i managed to get the new certificate going to this pass i still get 404
should i see the test file?

You only need to modify the first server block, the other one should remain as you posted in pastebin.

From this:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name chat.giladsky.com;

    location ^~ /.well-known/acme-challenge/ {
       default_type "text/plain";
       root         /usr/share/jitsi-meet;
    }
    location = /.well-known/acme-challenge/ {
       return 404;
    }
    location / {
       return 301 https://$host$request_uri;
    }


}

to this:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name chat.giladsky.com;

    location = /.well-known/acme-challenge/ {
       root /var/www/letsencrypt;
    }
    location / {
       return 301 https://$host$request_uri;
    }


}

Once done, reload nginx and try to get the test file again.

@giladsky

and now everything is working well!

I see you get your certificates but you are not using them in nginx, you must reload your nginx so it can use the new certificates. Tell me what is the command you should use to reload nginx (systemctl reload nginx or service nginx reload or ...) so we can include it in your acme.sh conf and the reload would be performed automatically in every renew.