Can not renew cert with header CSP


#1

My domain is: blog.stephane-huc.net

I ran this command:

acme-client -v blog.stephane-huc.net

It produced this output:

$ acme-client -v blog.stephane-huc.net
acme-client: /etc/ssl/acme/blog.stephane-huc.net.cert.pem: certificate renewable: 28 days left
acme-client: https://acme-v01.api.letsencrypt.org/directory: directories
acme-client: acme-v01.api.letsencrypt.org: DNS: 23.38.4.37
acme-client: https://acme-v01.api.letsencrypt.org/acme/new-authz: req-auth: blog.stephane-huc.net
acme-client: /var/www/acme/blog.stephane-huc.net/.well-known/acme-challenge/I8tex7T0VtWRAI1MLXFWSjbWDi3OCica7T_jz2euZ68: created
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/vhoFq1oFHf-SDg4vn8GLB7YA_E1YjNYTPJomaXbsC9A/1946384045: challenge
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/vhoFq1oFHf-SDg4vn8GLB7YA_E1YjNYTPJomaXbsC9A/1946384045: status
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/vhoFq1oFHf-SDg4vn8GLB7YA_E1YjNYTPJomaXbsC9A/1946384045: bad response
acme-client: transfer buffer: [{ "type": "http-01", "status": "invalid", "error": { "type": "urn:acme:error:connection", "detail": "Fetching http://blog.stephane-huc.net/.well-known/acme-challenge/I8tex7T0VtWRAI1MLXFWSjbWDi3OCica7T_jz2euZ68: Error getting validation data", "status": 400 }, "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/vhoFq1oFHf-SDg4vn8GLB7YA_E1YjNYTPJomaXbsC9A/1946384045", "token": "I8tex7T0VtWRAI1MLXFWSjbWDi3OCica7T_jz2euZ68", "keyAuthorization": "I8tex7T0VtWRAI1MLXFWSjbWDi3OCica7T_jz2euZ68.btIkQ8owertOE1LvXr1mezl9i5h6KptZrzIehfgwdcg", "validationRecord": [ { "url": "http://blog.stephane-huc.net/.well-known/acme-challenge/I8tex7T0VtWRAI1MLXFWSjbWDi3OCica7T_jz2euZ68", "hostname": "blog.stephane-huc.net", "port": "80", "addressesResolved": [ "213.246.39.160", "2a00:c70:1:213:246:39:160:1" ], "addressUsed": "2a00:c70:1:213:246:39:160:1", "addressesTried": [] } ] }] (971 bytes)
acme-client: bad exit: netproc(73533): 1

My web server is (include version):

nginx version: nginx/1.12.1
built with LibreSSL 2.5.2

The operating system my web server runs on is (include version): OpenBSD 6.1

My hosting provider, if applicable, is: Ikoula.fr

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no


Hi, i attempt to explain the problem.
I’ve few domains and subdomains managed into this server.
Quasi-all renew correctly, except both subdomains : ‘blog.stephane-huc.net’ and ‘ecrits.stephane-huc.net’.

I verify all rights folder, users rights on www and acme directories. No change. No problem recognized.
I got the idea to comment the header CSP : After, restart nginx, and retry with acme-client : i was able to renew my certs for my blog.

My CSP is :

content-security-policy: default-src 'none'; block-all-mixed-content; connect-src 'self' http://fontawesome.io; child-src 'self'; font-src 'self' data: https://cdn.rawgit.com https://fonts.gstatic.com https://maxcdn.bootstrapcdn.com ; form-action 'self';   frame-ancestors 'none'; img-src 'self' data: https://gravatar.com; report-uri https://hucste.report-uri.io/r/default/csp/reportOnly; reflected-xss block;sandbox allow-forms allow-modals allow-same-origin allow-scripts; script-src 'self' https://.disqus.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline' ; style-src 'self' https://cdn.rawgit.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline';

Header CSP for my website ecrits.stephane-huc.net:

content-security-policy: default-src 'none'; block-all-mixed-content; connect-src 'self' http://fontawesome.io; child-src 'self'; font-src 'self' data: https://cdn.rawgit.com https://fonts.googleapis.comfonts.googleapis.com https://fonts.gstatic.com https://maxcdn.bootstrapcdn.com ; form-action 'self'; frame-ancestors 'none'; img-src 'self' data: https://gravatar.com; report-uri https://hucste.report-uri.io/r/default/csp/reportOnly; reflected-xss block; referrer same-origin; sandbox allow-forms allow-modals allow-same-origin allow-scripts; script-src 'self' https://.disqus.com https://cdnjs.cloudflare.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline' ; style-src 'self' https://cdn.rawgit.com https://cdnjs.cloudflare.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline';

Cert no renew!


I read the explainations of @mnordhoff.
But, it seems really this CSP declaration prevents renew certs!


What’s wrong in my CSP declaration to permit renew?
(i have 28 days to resolve this…) :wink:


#2

LE prefers IPv6 addresses.
Can you reach your site via IPv6?


#3

Yes, normally, it can. All domains, in the server, are reach on IPv4+6 (*.stephane-huc.net, *.obsd4a.net)
It’s not the problem!
https://nat64check.ipv6-lab.net/v6score/measurement-21982/

In fact, if the header CSP commented, the certs are renewed.


#4

Ok, then I would look into these parts of the CSP:
connect-src
script-src


#5

Let’s Encrypt does not interpret CSP in any way.

It is possible that restarting the server (after you changed the CSP directive) coincidentally also solved whatever issue caused the first validation to fail. It might have also been some temporary network problem.

If you can reliably reproduce this by adding the original CSP again, then it might be related, though it would not necessarily be a CSP problem - perhaps you’re just exceeding some maximum total header size or something like that (I don’t know if that’s a thing).

Otherwise, I would assume this was some temporary issue. Note that you would have to get a new ACME account (or use a different domain) for each test; otherwise, existing valid authorizations will be reused automatically.


#6

@pfg: Hi, I know Let’s Encrypt not intercept CSP. It’s the opposite: CSP may block to communicate with LE!

Are you sure you understand how run CSP?!
(i dont want to offend you… )


@rg305: yes, i think it’s possible :wink:
I see that… and hope found a viable solution.


#7

I believe you are wrong… Content-Security-Policy is an ordinary header (just like Host, Referer or User-Agent…). It is meant to be interpreted by compatible web browsers - CSP itself is not able to block anything if it is not supported on the client side of HTTP connection. In fact, CSP simply does not apply to Let’s Encrypt VA servers, as they are not downloading any external resources included in the page content and they are not executing JavaScript. They care only about raw source of HTTP resource (and HTTP status codes - errors and redirects) - for Let’s Encrypt VA server something like <script>alert('1');</script> is just a string consisting of 28 characters, not a script.

I think the problem may be related to the length of your CSP header, not its contents - it’s quite long (over 700 bytes). Maybe there are some restrictions of response total size (headers + file contents) at Let’s Encrypt VA servers to prevent DoSing them - as response to the challenge is expected to be small.


#8

@mkwm: i know about CSP. And you are reason to recall me about needed compatible web browsers, even it’s deploy by nginx (in my case).

Interesting!
Whynot… :frowning:


#9

I have worked on a number of projects that use CSP. It is interpreted by and implemented in the user agent, which is Let’s Encrypt in this case - and Let’s Encrypt, unlike your browser, does not support CSP.

CSP does not do any kind of blocking on your server. From your server’s point of view, CSP is just yet another HTTP header. I don’t want to rule out something like the length of your headers causing some of this, but you’re with very, very high likelihood not looking at a CSP-related problem.


#10

Hi @hucste,

I’ve just installed OpenBSD 6.1 in a VM with IPv4 and IPv6 addresses, nginx 1.10.2 (using LibreSSL) and your CSP header:

add_header Content-Security-Policy "default-src 'none'; block-all-mixed-content; connect-src 'self' http://fontawesome
.io; child-src 'self'; font-src 'self' data: https://cdn.rawgit.com https://fonts.googleapis.comfonts.googleapis.com h
ttps://fonts.gstatic.com https://maxcdn.bootstrapcdn.com ; form-action 'self'; frame-ancestors 'none'; img-src 'self'
data: https://gravatar.com; report-uri https://hucste.report-uri.io/r/default/csp/reportOnly; reflected-xss block; ref
errer same-origin; sandbox allow-forms allow-modals allow-same-origin allow-scripts; script-src 'self' https://.disqus
.com https://cdnjs.cloudflare.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-in
line' ; style-src 'self' https://cdn.rawgit.com https://cdnjs.cloudflare.com https://fonts.googleapis.com https://maxc
dn.bootstrapcdn.com data: https: 'unsafe-inline';";

And I had no problem to issue a new certificate and renew it (I’ve forced it) with acme-client for domain bsd.sahsanu.com. So the problem should be in another side, maybe it was just a coincidence, who knows.

If you post the output of acme-client -n and the server blocks used in nginx for your domain (both, for port 80 and port 443) I could try to reproduce it.

Cheers,
sahsanu


#11

I understand you.
But, really, i was able to renew only when commenting on the header CSP.
It’s the fact!

The header CSP for my website stephane-huc.net not prevents me to renew, logically.


For the output of acme-client and server blocks used, can i send it you by private “message”?


#12

Hi @hucste,

Sure, you can send it to me via private message but if you share it here more buddies could help to you.

Cheers,
sahsanu


#13
$ acme-client -n
authority letsencrypt {
        agreement url "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
        api url "https://acme-v01.api.letsencrypt.org/directory"
        account key "/etc/acme/letsencrypt-privkey.pem"
}

authority letsencrypt-staging {
        agreement url "https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf"
        api url "https://acme-staging.api.letsencrypt.org/directory"
        account key "/etc/acme/letsencrypt-staging-privkey.pem"
}

domain stephane-huc.net {
        alternative names {  www.stephane-huc.net}
        domain key "/etc/ssl/acme/private/stephane-huc.net.privkey.pem"
        domain certificate "/etc/ssl/acme/stephane-huc.net.cert.pem"
        domain chain certificate "/etc/ssl/acme/stephane-huc.net.chain.pem"
        domain full chain certificate "/etc/ssl/acme/stephane-huc.net.fullchain.pem"
        sign with "letsencrypt"
        challengedir "/var/www/acme/stephane-huc.net/.well-known/acme-challenge"
}

domain blog.stephane-huc.net {
        domain key "/etc/ssl/acme/private/blog.stephane-huc.net.privkey.pem"
        domain certificate "/etc/ssl/acme/blog.stephane-huc.net.cert.pem"
        domain chain certificate "/etc/ssl/acme/blog.stephane-huc.net.chain.pem"
        domain full chain certificate "/etc/ssl/acme/blog.stephane-huc.net.fullchain.pem"
        sign with "letsencrypt"
        challengedir "/var/www/acme/blog.stephane-huc.net/.well-known/acme-challenge"
}

domain ecrits.stephane-huc.net {
        domain key "/etc/ssl/acme/private/ecrits.stephane-huc.net.privkey.pem"
        domain certificate "/etc/ssl/acme/ecrits.stephane-huc.net.cert.pem"
        domain chain certificate "/etc/ssl/acme/ecrits.stephane-huc.net.chain.pem"
        domain full chain certificate "/etc/ssl/acme/ecrits.stephane-huc.net.fullchain.pem"
        sign with "letsencrypt"
        challengedir "/var/www/acme/ecrits.stephane-huc.net/.well-known/acme-challenge"
}


usage: acme-client [-ADFnrv] [-f configfile] domain

I delete all informations for the domain ‘obsd4a.net:wink:


Server block nginx (only for subdomain ‘blog.stephane-huc.net’):

server {
    listen 80;
	listen [::]:80;	

	server_name blog.stephane-huc.net;

    include /etc/nginx/dir_blog.stephane-huc.net/headers.cfg;
    include /etc/nginx/dir_blog.stephane-huc.net/well_known.cfg;

    return 301 https://$server_name$request_uri;
}
server {
    
    listen 443 ssl http2;
	listen [::]:443 ssl http2;

	ssl on;

    ###########
    ### SSL cfguration
    ###########
    include /etc/nginx/dir_blog.stephane-huc.net/ssl.cfg;

    server_name blog.stephane-huc.net;

    root /www_dir_mad.stephane-huc.net/;

    access_log  /dir_logs/blog.stephane-huc.net/access.log compression if=$loggable;
    error_log   /dir_logs/blog.stephane-huc.net/errors.log info;

    include /etc/nginx/dir_blog.stephane-huc.net/headers.cfg;

    ##########
    ### Manage returns errors
    ##########
    include /etc/nginx/dir_blog.stephane-huc.net/return_403.cfg;
    include /etc/nginx/dir_blog.stephane-huc.net/return_444.cfg;
    include /etc/nginx/dir_blog.stephane-huc.net/return_4xx_Perishable_Press.cfg;

    ##########
    ### locations
    ##########
    location / {

        try_files $uri $uri/ @dokuwiki;

    }

    location ~ /(bin|conf|data|inc)/ {
        deny all;
    }

    location @dokuwiki {
        rewrite ^/_media/(.*)   /lib/exe/fetch.php?media=$1     last;
        rewrite ^/_detail/(.*)  /lib/exe/detail.php?media=$1    last;
        rewrite ^/_export/([^/]+)/(.*)  /doku.php?do=export_$1&id=$2     last;
        rewrite ^/tag/(.*)      /doku.php?id=tag:$1&do=showtag&tag=tag:$1        last;
        rewrite ^/(.*)          /doku.php?id=$1&$args    last;
    }

    location ~ \.php$ {
        try_files $uri =404;

        fastcgi_pass    unix:/run/php-fpm.sock;

        fastcgi_index  index.php;
    }


    # auth .well-known
    include /etc/nginx/dir_blog.stephane-huc.net/well_known.cfg;

    # manage errors
    include /etc/nginx/dir_blog.stephane-huc.net/mng_errors.cfg;

    # deny access to .htaccess files
    include /etc/nginx/dir_blog.stephane-huc.net/htaccess.cfg;

    # manage favicon
    include /etc/nginx/dir_blog.stephane-huc.net/favicon.cfg;

    # manage images
    include /etc/nginx/dir_blog.stephane-huc.net/img.cfg;

    # manage robots.txt
    include /etc/nginx/dir_blog.stephane-huc.net/robots.cfg;

    # manage scripts CSS/JS
    include /etc/nginx/dir_blog.stephane-huc.net/scripts.cfg;

}

file ‘well-known.cfg’:

location ^~ '/.well-known/acme-challenge' {
        allow all;
        root /var/www/acme/blog.stephane-huc.net/;
}

file ‘headers.cfg’:

add_header Access-Control-Allow-Origin "blog.stephane-huc.net" always;
add_header Content-Security-Policy "default-src 'none'; block-all-mixed-content;
 connect-src 'self' http://fontawesome.io; child-src 'self';
 font-src 'self' data: https://cdn.rawgit.com https://fonts.gstatic.com https://maxcdn.bootstrapcdn.com ; form-action 'self'; 
 frame-ancestors 'none'; img-src 'self' data: https://gravatar.com; 
 report-uri https://hucste.report-uri.io/r/default/csp/reportOnly; reflected-xss block; 
 sandbox allow-forms allow-modals allow-same-origin allow-scripts;
 script-src 'self' https://.disqus.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline' ; 
 style-src 'self' https://cdn.rawgit.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline';" always;
add_header Strict-Transport-Security "max-age=31536000;" always;
add_header Referrer-Policy "same-origin" always;
# manage X-Content-Type-Options: only one option "nosniff"
add_header X-Content-Type-Options "nosniff" always;
# manage X-Download-Options
# manage Frame: DENY, SAMEORIGIN or ALLOW-FROM https://example.com/
add_header Frame-Options "SAMEORIGIN" always;
add_header X-Frame-Options "SAMEORIGIN" always;
# manage Server information
add_header X-Powered-By "Blog: DCd 'n Powerd by Me, Myself, and moults softs :D" always;
# manage Robots
add_header X-Robots-Tag "index, nofollow";
# manage XSS Protection: 0, 1, or "1; mode=block" to block attacks rather than sanitising script
add_header X-Xss-Protection "1; mode=block" always;
# manage x-connection-hash
# manage x-response-time
# manage x-transaction
# manage x-ua-compatible
add_header x-ua-compatible "IE=edge,chrome=1";
# manage X-Permitted-Cross-Domain-Policies: none, master-only, by-content-type, by-ftp-filename, all
### 'master-only'     : master policy file allowed
### 'by-content-stype': served with 'Content-Type: text/x-cross-domain-policy' for HTTP/HTTPS only
### 'by-ftp-filename' : file names into crossdomain.xml allowed
add_header X-Permitted-Cross-Domain-Policies "none";


Same schema for website ‘ecrits.stephane-huc.net’!


Voila!


#14

Hi @hucste,

I’ve been playing with your conf and I had no problem to issue a new cert nor force a renew with your conf (headers included) so I don’t think the CSP headers are an issue to Let’s Encrypt to validate your domain.

Just 2 things:

You posted this CSP header:

add_header Content-Security-Policy "default-src 'none'; block-all-mixed-content;
 connect-src 'self' http://fontawesome.io; child-src 'self';
 font-src 'self' data: https://cdn.rawgit.com https://fonts.gstatic.com https://maxcdn.bootstrapcdn.com ; form-action 'self'; 
 frame-ancestors 'none'; img-src 'self' data: https://gravatar.com; 
 report-uri https://hucste.report-uri.io/r/default/csp/reportOnly; reflected-xss block; 
 sandbox allow-forms allow-modals allow-same-origin allow-scripts;
 script-src 'self' https://.disqus.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline' ; 
 style-src 'self' https://cdn.rawgit.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline';" always;

And the content is separated in several lines when it should be just one, maybe it was just an issue when you pasted it here because accesing your site seems it is only one line.

Before I realized this issue I was able to issue a cert but firefox refused to connect to the site, when I fixed the header issue I was able to renew the cert and firefox connected pretty fine, so, as I said, the headers are not an issue here to be able to validate your domain.

The other thing you should remove in your server block for port 80 is this header:

add_header Strict-Transport-Security "max-age=31536000;" always;

This header should be added ONLY in https server block.

Anyway, that is not a problem to be able to issue a cert too.

I’ve been playing a few hours with the conf you passed and I see no problem, at least I had none.

Maybe one of our colleagues can see something evident but sincerely I think your conf was not an issue to renew your cert.

Cheers,
sahsanu


#15

Ok, thanks.
It really gets me down not to be believed, especially as I am a technician computer.
But i know what i’m doing: I can renew certs for my bot subdomains ‘blog’ and ‘ecrits’ only if i comment header CSP.

HSTS maybe poorly written… i will fix for only HTTPS; but this not prevents to renew certs.

Ty!


#16

I’m not saying I don’t believe you, just saying that with your nginx configuration I was able to issue and renew the certs so I couldn’t reproduce your problem :frowning: maybe it is a combination of several and specific factors related to your whole conf…

Maybe @cpu or @jsha could confirm whether the configured headers could be an issue in certain circumstances, in my case they weren’t.

I hope someone could give to you an explanation because honestly, I can’t.

Good luck,
sahsanu


#17

Where do you read this?
Nowhere on Docs CSP W3C:

The Content-Security-Policy HTTP response header field is the preferred mechanism for delivering a policy from a server to a client. The header’s value is represented by the following ABNF [RFC5234]:

And if i read correctly the RFC 7231, particularly the subchapter “Syntax Notation”, and “Appendice D - Collected ABNF” - pages 94,95, you see few examples in several lines.
Egual on RFC 5234, subchapter “ABNF Definition of ABNF” or “Appendix B. Core ABNF of ABNF”:

(…)
rule = rulename defined-as elements c-nl
; continues if next line starts
; with white space
(…)
defined-as = *c-wsp ("=" / “=/”) *c-wsp
; basic rules definition and
; incremental alternatives
(…)

Works CSP is based on both RFC, and others…

I’m wrong?

as you can read, my CSP is box by double-quotes :wink:


#18

I didn’t read it in anywhere, this conf mades my Firefox browser not able to access the site:

add_header Content-Security-Policy "default-src 'none'; block-all-mixed-content;
 connect-src 'self' http://fontawesome.io; child-src 'self';
 font-src 'self' data: https://cdn.rawgit.com https://fonts.gstatic.com https://maxcdn.bootstrapcdn.com ; form-action 'self'; 
 frame-ancestors 'none'; img-src 'self' data: https://gravatar.com; 
 report-uri https://hucste.report-uri.io/r/default/csp/reportOnly; reflected-xss block; 
 sandbox allow-forms allow-modals allow-same-origin allow-scripts;
 script-src 'self' https://.disqus.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline' ; 
 style-src 'self' https://cdn.rawgit.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline';" always;

This one works as expected:

add_header Content-Security-Policy "default-src 'none'; block-all-mixed-content; connect-src 'self' http://fontawesome.io; child-src 'self'; font-src 'self' data: https://cdn.rawgit.com https://fonts.gstatic.com https://maxcdn.bootstrapcdn.com ; form-action 'self'; frame-ancestors 'none'; img-src 'self' data: https://gravatar.com; report-uri https://hucste.report-uri.io/r/default/csp/reportOnly; reflected-xss block; sandbox allow-forms allow-modals allow-same-origin allow-scripts; script-src 'self' https://.disqus.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline' ; style-src 'self' https://cdn.rawgit.com https://fonts.googleapis.com https://maxcdn.bootstrapcdn.com data: https: 'unsafe-inline';" always;

Maybe it was just an issue copying your conf from the forum. When I try to reproduce a problem I must copy and paste the conf provided, I’m not wiiling to think if there are spaces, tabs or null characters in front of each line so the header is correct or it isn’t because I’m trying to reproduce a specific issue with a web server conf so, every detail matters. I’ve found this issue that was solved putting the header contents in just one line and I’m telling it to you just in case you could have the same issue, if your conf works, perfect, it didn’t work on my server.

Cheers,
sahsanu


#19

I run CSP on all of my sites, never had one issue ever with a CSP policy affecting an issue or renewal of a certificate. I log all CSP failures so I would have seen this but never seen anything from let’s encrypt every show up in my CSP violations.


#20

I think @sahsanu is on the right track here with the idea that your CSP header is malformed. Sometimes when Boulder returns a generic error to the user, a more detailed one is logged on the server side. I looked for this one, and found:

Get http://blog.stephane-huc.net/.well-known/acme-challenge/I8tex7T0VtWRAI1MLXFWSjbWDi3OCica7T_jz2euZ68: malformed MIME header line: form-action ‘self’;

It looks like you were printing form-action 'self'; on a line by itself, which is an invalid HTTP response (interesting that Nginx is willing to send it!).