Invalid Response - Verify Error 404

I moved my self hosted nextcloud server from a domain to another and try to issue a new certificate.
Generally I use the installation-settings from here [German]: Nextcloud Installationsanleitung - Carsten Rieger IT-Services

I ran this command with the user acmeuser:

acme.sh --issue -d redacted.domain.tld --server letsencrypt --keylength 4096 -w /var/www/letsencrypt --key-file /etc/letsencrypt/rsa-certs/privkey.pem --ca-file /etc/letse
ncrypt/rsa-certs/chain.pem --cert-file /etc/letsencrypt/rsa-certs/cert.pem --fullchain-file /etc/letsencrypt/rsa-certs/fullchain.pem

It produced this output:

[Mon 24 Apr 2023 09:06:17 AM CEST] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Mon 24 Apr 2023 09:06:17 AM CEST] Single domain='redacted.domain.tld'
[Mon 24 Apr 2023 09:06:17 AM CEST] Getting domain auth token for each domain
[Mon 24 Apr 2023 09:06:19 AM CEST] Getting webroot for domain='redacted.domain.tld'
[Mon 24 Apr 2023 09:06:19 AM CEST] Verifying: redacted.domain.tld
[Mon 24 Apr 2023 09:06:19 AM CEST] Pending, The CA is processing your order, please just wait. (1/30)
[Mon 24 Apr 2023 09:06:23 AM CEST] redacted.domain.tld:Verify error:MY.PUBLIC.IP: Invalid response from https://redacted.domain.tld/.well-known/acme-challenge/saaMJrmrqasCGiGr0d1_FiydnrE_NnAYQPSBUYPG1-0: 404

[Mon 24 Apr 2023 08:58:12 AM CEST] Running cmd: setdefaultca
[Mon 24 Apr 2023 08:58:12 AM CEST] Changed default CA to: ^[[1;32mhttps://acme-v02.api.letsencrypt.org/directory^[[0m
[Mon 24 Apr 2023 08:58:36 AM CEST] Running cmd: issue
[Mon 24 Apr 2023 08:58:36 AM CEST] _main_domain='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:36 AM CEST] _alt_domains='no'
[Mon 24 Apr 2023 08:58:36 AM CEST] Using config home:/home/acmeuser/.acme.sh
[Mon 24 Apr 2023 08:58:36 AM CEST] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Mon 24 Apr 2023 08:58:36 AM CEST] DOMAIN_PATH='/home/acmeuser/.acme.sh/redacted.domain.tld'
[Mon 24 Apr 2023 08:58:36 AM CEST] Le_NextRenewTime
[Mon 24 Apr 2023 08:58:36 AM CEST] Using ACME_DIRECTORY: https://acme-v02.api.letsencrypt.org/directory
[Mon 24 Apr 2023 08:58:36 AM CEST] _init api for server: https://acme-v02.api.letsencrypt.org/directory
[Mon 24 Apr 2023 08:58:36 AM CEST] GET
[Mon 24 Apr 2023 08:58:36 AM CEST] url='https://acme-v02.api.letsencrypt.org/directory'
[Mon 24 Apr 2023 08:58:36 AM CEST] timeout=
[Mon 24 Apr 2023 08:58:36 AM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 08:58:37 AM CEST] ret='0'
[Mon 24 Apr 2023 08:58:37 AM CEST] ACME_KEY_CHANGE='https://acme-v02.api.letsencrypt.org/acme/key-change'
[Mon 24 Apr 2023 08:58:37 AM CEST] ACME_NEW_AUTHZ
[Mon 24 Apr 2023 08:58:37 AM CEST] ACME_NEW_ORDER='https://acme-v02.api.letsencrypt.org/acme/new-order'
[Mon 24 Apr 2023 08:58:37 AM CEST] ACME_NEW_ACCOUNT='https://acme-v02.api.letsencrypt.org/acme/new-acct'
[Mon 24 Apr 2023 08:58:37 AM CEST] ACME_REVOKE_CERT='https://acme-v02.api.letsencrypt.org/acme/revoke-cert'
[Mon 24 Apr 2023 08:58:37 AM CEST] ACME_AGREEMENT='https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf'
[Mon 24 Apr 2023 08:58:37 AM CEST] ACME_NEW_NONCE='https://acme-v02.api.letsencrypt.org/acme/new-nonce'
[Mon 24 Apr 2023 08:58:37 AM CEST] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Mon 24 Apr 2023 08:58:37 AM CEST] _on_before_issue
[Mon 24 Apr 2023 08:58:37 AM CEST] _chk_main_domain='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:37 AM CEST] _chk_alt_domains
[Mon 24 Apr 2023 08:58:37 AM CEST] Le_LocalAddress
[Mon 24 Apr 2023 08:58:37 AM CEST] d='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:37 AM CEST] Check for domain='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:37 AM CEST] _currentRoot='/var/www/letsencrypt'
[Mon 24 Apr 2023 08:58:37 AM CEST] d
[Mon 24 Apr 2023 08:58:37 AM CEST] _saved_account_key_hash is not changed, skip register account.
[Mon 24 Apr 2023 08:58:37 AM CEST] Read key length:4096
[Mon 24 Apr 2023 08:58:37 AM CEST] _createcsr
[Mon 24 Apr 2023 08:58:37 AM CEST] Single domain='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:37 AM CEST] Getting domain auth token for each domain
[Mon 24 Apr 2023 08:58:37 AM CEST] d
[Mon 24 Apr 2023 08:58:37 AM CEST] url='https://acme-v02.api.letsencrypt.org/acme/new-order'
[Mon 24 Apr 2023 08:58:37 AM CEST] payload='{"identifiers": [{"type":"dns","value":"redacted.domain.tld"}]}'
[Mon 24 Apr 2023 08:58:37 AM CEST] RSA key
[Mon 24 Apr 2023 08:58:37 AM CEST] HEAD
[Mon 24 Apr 2023 08:58:37 AM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/new-nonce'
[Mon 24 Apr 2023 08:58:37 AM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g  -I  '
[Mon 24 Apr 2023 08:58:38 AM CEST] _ret='0'
[Mon 24 Apr 2023 08:58:38 AM CEST] code='201'
[Mon 24 Apr 2023 08:58:38 AM CEST] Le_LinkOrder='https://acme-v02.api.letsencrypt.org/acme/order/349339550/178067177717'
[Mon 24 Apr 2023 08:58:38 AM CEST] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/349339550/178067177717'
[Mon 24 Apr 2023 08:58:38 AM CEST] url='https://acme-v02.api.letsencrypt.org/acme/authz-v3/222056760747'
[Mon 24 Apr 2023 08:58:38 AM CEST] payload
[Mon 24 Apr 2023 08:58:38 AM CEST] POST
[Mon 24 Apr 2023 08:58:38 AM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/authz-v3/222056760747'
[Mon 24 Apr 2023 08:58:38 AM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 08:58:38 AM CEST] _ret='0'
[Mon 24 Apr 2023 08:58:38 AM CEST] code='200'
[Mon 24 Apr 2023 08:58:38 AM CEST] d='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:38 AM CEST] Getting webroot for domain='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:38 AM CEST] _w='/var/www/letsencrypt'
[Mon 24 Apr 2023 08:58:38 AM CEST] _currentRoot='/var/www/letsencrypt'
[Mon 24 Apr 2023 08:58:38 AM CEST] entry='"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw","token":"WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU"'
[Mon 24 Apr 2023 08:58:38 AM CEST] token='WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU'
[Mon 24 Apr 2023 08:58:38 AM CEST] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:38 AM CEST] keyauthorization='WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ'
[Mon 24 Apr 2023 08:58:38 AM CEST] dvlist='redacted.domain.tld#WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ#https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw#http-01#/va>
[Mon 24 Apr 2023 08:58:38 AM CEST] d
[Mon 24 Apr 2023 08:58:38 AM CEST] vlist='redacted.domain.tld#WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ#https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw#http-01#/var>
[Mon 24 Apr 2023 08:58:38 AM CEST] d='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:38 AM CEST] ok, let's start to verify
[Mon 24 Apr 2023 08:58:38 AM CEST] Verifying: redacted.domain.tld
[Mon 24 Apr 2023 08:58:38 AM CEST] d='redacted.domain.tld'
[Mon 24 Apr 2023 08:58:38 AM CEST] keyauthorization='WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ'
[Mon 24 Apr 2023 08:58:38 AM CEST] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:38 AM CEST] _currentRoot='/var/www/letsencrypt'
[Mon 24 Apr 2023 08:58:38 AM CEST] wellknown_path='/var/www/letsencrypt/.well-known/acme-challenge'
[Mon 24 Apr 2023 08:58:38 AM CEST] writing token:WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU to /var/www/letsencrypt/.well-known/acme-challenge/WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU
[Mon 24 Apr 2023 08:58:38 AM CEST] Changing owner/group of .well-known to www-data:www-data
[Mon 24 Apr 2023 08:58:38 AM CEST] chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/text.txt': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known': Operation not permitted
[Mon 24 Apr 2023 08:58:38 AM CEST] chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/text.txt': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known': Operation not permitted
[Mon 24 Apr 2023 08:58:38 AM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:38 AM CEST] payload='{}'
[Mon 24 Apr 2023 08:58:38 AM CEST] POST
[Mon 24 Apr 2023 08:58:38 AM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:38 AM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 08:58:39 AM CEST] _ret='0'
[Mon 24 Apr 2023 08:58:39 AM CEST] code='200'
[Mon 24 Apr 2023 08:58:39 AM CEST] trigger validation code: 200
[Mon 24 Apr 2023 08:58:39 AM CEST] Pending, The CA is processing your order, please just wait. (1/30)
[Mon 24 Apr 2023 08:58:39 AM CEST] sleep 2 secs to verify again
[Mon 24 Apr 2023 08:58:42 AM CEST] checking
[Mon 24 Apr 2023 08:58:42 AM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:42 AM CEST] payload
[Mon 24 Apr 2023 08:58:42 AM CEST] POST
[Mon 24 Apr 2023 08:58:42 AM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:42 AM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 08:58:42 AM CEST] _ret='0'
[Mon 24 Apr 2023 08:58:42 AM CEST] code='200'
[Mon 24 Apr 2023 08:58:42 AM CEST] redacted.domain.tld:Verify error:MY.PUBLIC.IP: Invalid response from https://redacted.domain.tld/.well-known/acme-challenge/WN6rYSdyauNulLNLLJnZstFMqGHSpXOQPY5fcnuMYiU: 404
[Mon 24 Apr 2023 08:58:42 AM CEST] pid
[Mon 24 Apr 2023 08:58:42 AM CEST] No need to restore nginx, skip.
[Mon 24 Apr 2023 08:58:42 AM CEST] _clearupdns
[Mon 24 Apr 2023 08:58:42 AM CEST] dns_entries
[Mon 24 Apr 2023 08:58:42 AM CEST] skip dns.
[Mon 24 Apr 2023 08:58:42 AM CEST] _on_issue_err
[Mon 24 Apr 2023 08:58:42 AM CEST] Please check log file for more details: /home/acmeuser/.acme.sh/acme.sh.log
[Mon 24 Apr 2023 08:58:42 AM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:42 AM CEST] payload='{}'
[Mon 24 Apr 2023 08:58:43 AM CEST] POST
[Mon 24 Apr 2023 08:58:43 AM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222056760747/bGFXjw'
[Mon 24 Apr 2023 08:58:43 AM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 08:58:43 AM CEST] _ret='0'
[Mon 24 Apr 2023 08:58:43 AM CEST] code='400'

If I change the owner of /var/www/letsencrypt to acmeuser instead of www-data, the chown errors disappear (see chown: changing ownership not permitted · Issue #163 · acmesh-official/acme.sh · GitHub ), but the 404 stays. If I place a test.txt in the acme-challenge folder I also get a 404 when I try to access https://redacted.domain.tld/.well-known/acme-challenge/test.txt. The redacted.domain.tld is accessible with the self signed certificate.

nginx -T:

nginx: [warn] "ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/ssl/certs/ssl-cert-snakeoil.pem"
nginx: [warn] conflicting server name "redacted.domain.tld" on 0.0.0.0:80, ignored
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /var/run/nginx.pid;
events {
  worker_connections 2048;
  multi_accept on; use epoll;
  }
http {
  log_format xyzzz escape=json
  '{'
    '"time_local":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"request":"$request",'
    '"status": "$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';
  server_names_hash_bucket_size 64;
  access_log /var/log/nginx/access.log xyzzz;
  error_log /var/log/nginx/error.log warn;
  set_real_ip_from 192.168.1.135;    #########ip from reverse proxy
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;
  include /etc/nginx/mime.types;
  default_type application/octet-stream;
  sendfile on;
  send_timeout 3600;
  tcp_nopush on;
  tcp_nodelay on;
  open_file_cache max=500 inactive=10m;
  open_file_cache_errors on;
  keepalive_timeout 65;
  reset_timedout_connection on;
  server_tokens off;
  resolver 127.0.0.53 valid=30s;
  resolver_timeout 5s;
  include /etc/nginx/conf.d/*.conf;
  }

# configuration file /etc/nginx/mime.types:

types {
###########redacted to shorten file
}

# configuration file /etc/nginx/conf.d/default.conf:

# configuration file /etc/nginx/conf.d/http.conf:
upstream php-handler {
server unix:/run/php/php8.1-fpm.sock;
}
map $arg_v $asset_immutable {
"" "";
default "immutable";
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name redacted.domain.tld;
root /var/www;
location ^~ /.well-known/acme-challenge {
default_type text/plain;
root /var/www/letsencrypt;
}
location / {
return 301 https://$host$request_uri;
}
}

# configuration file /etc/nginx/conf.d/nextcloud.conf:
server {
listen 443      ssl http2;
listen [::]:443 ssl http2;
listen 80;
server_name redacted.domain.tld;
ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
ssl_trusted_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
#ssl_certificate /etc/letsencrypt/rsa-certs/fullchain.pem;
#ssl_certificate_key /etc/letsencrypt/rsa-certs/privkey.pem;
#ssl_certificate /etc/letsencrypt/ecc-certs/fullchain.pem;
#ssl_certificate_key /etc/letsencrypt/ecc-certs/privkey.pem;
#ssl_trusted_certificate /etc/letsencrypt/ecc-certs/chain.pem;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_protocols TLSv1.3 TLSv1.2;
ssl_ciphers 'TLS-CHACHA20-POLY1305-SHA256:TLS-AES-256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384';
ssl_ecdh_curve X448:secp521r1:secp384r1;
ssl_prefer_server_ciphers on;
ssl_stapling on;
ssl_stapling_verify on;
client_max_body_size 10G;
client_body_timeout 3600s;
client_body_buffer_size 512k;
fastcgi_buffers 64 4K;
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/wasm application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;
add_header Strict-Transport-Security            "max-age=15768000; includeSubDomains; preload;" always;
add_header Permissions-Policy                   "interest-cohort=()";
add_header Referrer-Policy                      "no-referrer"   always;
add_header X-Content-Type-Options               "nosniff"       always;
add_header X-Download-Options                   "noopen"        always;
add_header X-Frame-Options                      "SAMEORIGIN"    always;
add_header X-Permitted-Cross-Domain-Policies    "none"          always;
add_header X-Robots-Tag                         "noindex, nofollow" always;
add_header X-XSS-Protection                     "1; mode=block" always;
fastcgi_hide_header X-Powered-By;
root /var/www/nextcloud;
index index.php index.html /index.php$request_uri;
location = / {
if ( $http_user_agent ~ ^DavClnt ) {
return 302 /remote.php/webdav/$is_args$args;
}
}
location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}
location ^~ /.well-known {
location = /.well-known/carddav { return 301 /remote.php/dav/; }
location = /.well-known/caldav  { return 301 /remote.php/dav/; }
location /.well-known/acme-challenge { try_files $uri $uri/ =404; }
location /.well-known/pki-validation { try_files $uri $uri/ =404; }
return 301 /index.php$request_uri;
}
location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }
location ~ \.php(?:$|/) {
rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;
fastcgi_split_path_info ^(.+?\.php)(/.*)$;
set $path_info $fastcgi_path_info;
try_files $fastcgi_script_name =404;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $path_info;
fastcgi_param HTTPS on;
fastcgi_param modHeadersAvailable true;
fastcgi_param front_controller_active true;
fastcgi_pass php-handler;
fastcgi_intercept_errors on;
fastcgi_request_buffering off;
fastcgi_read_timeout 3600;
fastcgi_send_timeout 3600;
fastcgi_connect_timeout 3600;
fastcgi_max_temp_file_size 0;
}
location ~ \.(?:css|js|svg|gif|png|jpg|ico|wasm|tflite|map)$ {
try_files $uri /index.php$request_uri;
add_header Cache-Control "public, max-age=15778463, $asset_immutable";
expires 6M;
access_log off;
location ~ \.wasm$ {
default_type application/wasm;
}
}
location ~ \.woff2?$ {
try_files $uri /index.php$request_uri;
expires 7d;
access_log off;
}
location /remote {
return 301 /remote.php$request_uri;
}
location / {
try_files $uri $uri/ /index.php$request_uri;
}
}

# configuration file /etc/nginx/fastcgi_params:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  REQUEST_SCHEME     $scheme;
fastcgi_param  HTTPS              $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

The operating system my web server runs on is (include version):
Debian 11

https://crt.sh does not show any certs for redacted.domain.tld.

Thank you for any help. I really appreciate it!

does this show any messages about the request?

2 Likes
2023/04/24 08:35:30 [error] 22006#22006: *1 open() "/var/www/nextcloud/.well-known/acme-challenge/text.txt" failed (2: No such file or directory), client: 10.10.40.1, server: redacted.domain.tld, request: "GET /.well-known/acme-challenge/text.txt HTTP/2.0", host: "redacted.domain.tld"
2023/04/24 08:35:46 [error] 22006#22006: *1 open() "/var/www/nextcloud/.well-known/acme-challenge/text.txt" failed (2: No such file or directory), client: 10.10.40.1, server: redacted.domain.tld, request: "GET /.well-known/acme-challenge/text.txt HTTP/2.0", host: "redacted.domain.tld"
2023/04/24 08:37:40 [error] 22006#22006: *2 open() "/var/www/nextcloud/.well-known/acme-challenge/z1Te5ETFo02o4mNb-uw3aa8BM9LbKix2JZ9xyTdVYUA" failed (2: No such file or directory), client: 10.10.40.1, server: redacted.domain.tld, request: "GET /.well-known/acme-challenge/z1Te5ETFo02o4mNb-uw3aa8BM9LbKix2JZ9xyTdVYUA HTTP/1.1", host: "redacted.domain.tld", referrer: "http://redacted.domain.tld/.well-known/acme-challenge/z1Te5ETFo02o4mNb-uw3aa8BM9LbKix2JZ9xyTdVYUA"
2023/04/24 08:57:35 [warn] 24020#24020: "ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/ssl/certs/ssl-cert-snakeoil.pem"
2023/04/24 08:57:35 [warn] 24020#24020: conflicting server name "redacted.domain.tld" on 0.0.0.0:80, ignored
2023/04/24 09:03:16 [warn] 25251#25251: "ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/ssl/certs/ssl-cert-snakeoil.pem"
2023/04/24 09:03:16 [warn] 25251#25251: conflicting server name "redacted.domain.tld" on 0.0.0.0:80, ignored
2023/04/24 09:24:57 [warn] 27245#27245: "ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/ssl/certs/ssl-cert-snakeoil.pem"
2023/04/24 09:24:57 [warn] 27245#27245: conflicting server name "redacted.domain.tld" on 0.0.0.0:80, ignored

I don't think so?

As your error.log says, nginx tries to open a file in /var/www/nextcloud instead of /var/www/letsencrypt.

It appears that you have extraneous listen 80; in your nextcloud.conf. Try removing this line and try the acme.sh command again

3 Likes

Thanks for the answer. I forgot to delete my modification there, thanks for spotting it. It does not solve the 404 though.

1 Like

Did you restart nginx after that change?

Try a new request to your test.txt file and check the error log. What does it say?

By the way, this would go quicker if you did not redact your domain name.

4 Likes

Yes I did. Thank you for your help.
acme.sh.log still:

[Mon 24 Apr 2023 07:52:17 PM CEST] Le_LinkOrder='https://acme-v02.api.letsencrypt.org/acme/order/349339550/178156138057'
[Mon 24 Apr 2023 07:52:17 PM CEST] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/349339550/178156138057'
[Mon 24 Apr 2023 07:52:17 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/authz-v3/222181401917'
[Mon 24 Apr 2023 07:52:17 PM CEST] payload
[Mon 24 Apr 2023 07:52:17 PM CEST] POST
[Mon 24 Apr 2023 07:52:17 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/authz-v3/222181401917'
[Mon 24 Apr 2023 07:52:17 PM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 07:52:18 PM CEST] _ret='0'
[Mon 24 Apr 2023 07:52:18 PM CEST] code='200'
[Mon 24 Apr 2023 07:52:18 PM CEST] d='redacted.domain.tld'
[Mon 24 Apr 2023 07:52:18 PM CEST] Getting webroot for domain='redacted.domain.tld'
[Mon 24 Apr 2023 07:52:18 PM CEST] _w='/var/www/letsencrypt'
[Mon 24 Apr 2023 07:52:18 PM CEST] _currentRoot='/var/www/letsencrypt'
[Mon 24 Apr 2023 07:52:18 PM CEST] entry='"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA","token":"Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8"'
[Mon 24 Apr 2023 07:52:18 PM CEST] token='Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8'
[Mon 24 Apr 2023 07:52:18 PM CEST] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA'
[Mon 24 Apr 2023 07:52:18 PM CEST] keyauthorization='Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ'
[Mon 24 Apr 2023 07:52:18 PM CEST] dvlist='redacted.domain.tld#Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ#https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA#http-01#>
[Mon 24 Apr 2023 07:52:18 PM CEST] d
[Mon 24 Apr 2023 07:52:18 PM CEST] vlist='redacted.domain.tld#Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ#https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA#http-01#/>
[Mon 24 Apr 2023 07:52:18 PM CEST] d='redacted.domain.tld'
[Mon 24 Apr 2023 07:52:18 PM CEST] ok, let's start to verify
[Mon 24 Apr 2023 07:52:18 PM CEST] Verifying: redacted.domain.tld
[Mon 24 Apr 2023 07:52:18 PM CEST] d='redacted.domain.tld'
[Mon 24 Apr 2023 07:52:18 PM CEST] keyauthorization='Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8.SwePoNRE-hU6oefh2ALsL8J8dD23JwNV7QMozWzz6oQ'
[Mon 24 Apr 2023 07:52:18 PM CEST] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA'
[Mon 24 Apr 2023 07:52:18 PM CEST] _currentRoot='/var/www/letsencrypt'
[Mon 24 Apr 2023 07:52:18 PM CEST] wellknown_path='/var/www/letsencrypt/.well-known/acme-challenge'
[Mon 24 Apr 2023 07:52:18 PM CEST] writing token:Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8 to /var/www/letsencrypt/.well-known/acme-challenge/Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8
[Mon 24 Apr 2023 07:52:18 PM CEST] Changing owner/group of .well-known to acmeuser:www-data
[Mon 24 Apr 2023 07:52:18 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA'
[Mon 24 Apr 2023 07:52:18 PM CEST] payload='{}'
[Mon 24 Apr 2023 07:52:18 PM CEST] POST
[Mon 24 Apr 2023 07:52:18 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA'
[Mon 24 Apr 2023 07:52:18 PM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 07:52:19 PM CEST] _ret='0'
[Mon 24 Apr 2023 07:52:19 PM CEST] code='200'
[Mon 24 Apr 2023 07:52:19 PM CEST] trigger validation code: 200
[Mon 24 Apr 2023 07:52:19 PM CEST] Pending, The CA is processing your order, please just wait. (1/30)
[Mon 24 Apr 2023 07:52:19 PM CEST] sleep 2 secs to verify again
[Mon 24 Apr 2023 07:52:22 PM CEST] checking
[Mon 24 Apr 2023 07:52:22 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA'
[Mon 24 Apr 2023 07:52:22 PM CEST] payload
[Mon 24 Apr 2023 07:52:22 PM CEST] POST
[Mon 24 Apr 2023 07:52:22 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/222181401917/8fqqwA'
[Mon 24 Apr 2023 07:52:22 PM CEST] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Mon 24 Apr 2023 07:52:22 PM CEST] _ret='0'
[Mon 24 Apr 2023 07:52:22 PM CEST] code='200'
[Mon 24 Apr 2023 07:52:22 PM CEST] redacted.domain.tld:Verify error:EXTERNAL.IP: Invalid response from https://redacted.domain.tld/.well-known/acme-challenge/Grj38zz3oqY1LODX4CwQqZhwSAnI7SWX59moQIjsvQ8: 404

just this in error.log after trying to reach the file text.txt (nothing new, browser shows 404):

2023/04/24 19:52:08 [warn] 51082#51082: "ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/ssl/certs/ssl-cert-snakeoil.pem"

Is there another benefit of disclosing the domain except time?

Using this won't scale at all.

If you only need one single cert, then it may work for you.
But I'm still concerned on what may happen on renewals.

2 Likes

Did the 404 show in the access.log? What did that look like?

We are (mostly) volunteers here so we all value our time differently. For example, I don't spend much time on threads that are not progressing forward.

Without the domain name I can only offer general advice. If you are a very skilled server admin sometimes a few general clues are enough. But, otherwise the problem is not likely to be solved without letting us use our full toolkit.

4 Likes

Should I omit that all? I just need one certificate on this server normally.

@MikeMcQ
The 404 did not show in the access.log (except maybe this no OCSP responder URL). Maybe I should create the testfile with another user?

/var/www/letsencrypt# ls -l
total 4
-rw-r--r-- 1 acmeuser acmeuser 9 Apr 24 19:53 text.txt

Thanks for the explanation! And sorry for the delay... Sent you the domain name privately.

1 Like

Requests to your server should show up in the appropriate log.

If not the request isn't going to that server. Or, your log isn't setup right.

4 Likes

Hmm, I see a nginx 404 error when trying to access https://redacted.domain.tld/.well-known/acme-challenge/text.txt via browser. No error in the /var/log/nginx/error.log. So there should be an entry - I try to find out why there is no entry there, thanks!

1 Like

And, your access.log says?

4 Likes

Just found it:

{"time_local":"24/Apr/2023:21:43:19 +0200","remote_addr":"10.10.40.1","remote_user":"","request":"GET /.well-known/acme-challenge/text.txt HTTP/2.0","status": "404","body_bytes_sent":"146","request_time":"0.000","http_referrer":"","http_user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"}

The nginx -T output you showed earlier is not the one handling the HTTP Challenge requests.

I used the domain name you DM'd me and got redirected instead. That should not happen with the server block you showed.

curl -i http://[redacted]/.well-known/acme-challenge/MikeTest123
HTTP/1.1 302 Found
content-length: 0
location: https://[redacted]/.well-known/acme-challenge/MikeTest123
cache-control: no-cache

Had you shared your domain earlier this key info would have been obvious earlier.

4 Likes

I'm sorry!
Then it's a HAproxy problem, I will try to fix it.
Thanks a lot!

2 Likes

Important detail to have omitted.

2 Likes

Thank you for your help. :smiling_face_with_three_hearts:
Changing HAProxy to forward and not redirect HTTP (80) requests to HTTPS was the key.

I should keep Port 80 open according to this source: Best Practice - Keep Port 80 Open - Let's Encrypt

How do I curl properly from inside the network to see the same output like you did from outside the network?

@rg305 actually hidden in the nginx -T was my comment about a reverse proxy. Because I did not change anything there (or I thought so) I did not mention it.

2 Likes

Give the Port 80 comment I would think you would want to go from inside the network to the outside internet network and then back inside to test that Port 80 is not being blocked.

2 Likes

I don't think you can be outside the network while inside the network.

3 Likes