Lets-encrypt failing

Help
21

do you use .htaccess files?

22

Thanks for the info. I made an issue for this at mod_security can break the Apache plugin Ā· Issue #7456 Ā· certbot/certbot Ā· GitHub. If I understand the issue in general correctly, this will be a bit of a cat and mouse game for us in Apache because there is no way for Certbot to ensure the request is served how we want despite the presence of any module.

If people are aware of other issues or things we can do to work around them, I'd love to chat (probably outside of this thread).

I did the same thing using the latest version of Certbot in GitHub with the same result.

23

Thanks, are there other files that are being included by your Apache configuration? Maybe you can look for them with ls -lR /etc/httpd/conf.

24

-rw-r----- 1 root root 15353 Oct 7 12:36 1
-rw-r----- 1 root root 18492 Oct 18 17:52 httpd.conf
-rw-r----- 1 apache apache 18106 Oct 15 19:01 httpd.conf.bak
-rw-r----- 1 root root 18178 Oct 16 13:16 httpd_new.conf
-rw-r----- 1 root root 17491 Oct 9 12:14 httpd_updated.conf
-rw-r--r-- 1 root root 13077 Aug 8 06:42 magic
-rwxrwxr-x 1 apache apache 1752 Oct 16 00:15 shib.conf

25

try:
sudo apache2ctl -DDUMP_CONFIG

and again, do you use .htaccess files?

26

sudo httpd -DDUMP_CONFIG
[Fri Oct 18 18:14:42.454694 2019] [so:warn] [pid 210160] AH01574: module proxy_module is already loaded, skipping
[Fri Oct 18 18:14:42.469871 2019] [so:warn] [pid 210160] AH01574: module rewrite_module is already loaded, skipping
[Fri Oct 18 18:14:42.469910 2019] [so:warn] [pid 210160] AH01574: module proxy_module is already loaded, skipping
[Fri Oct 18 18:14:42.469956 2019] [so:warn] [pid 210160] AH01574: module proxy_http_module is already loaded, skipping
[Fri Oct 18 18:14:42.470155 2019] [so:warn] [pid 210160] AH01574: module headers_module is already loaded, skipping
[Fri Oct 18 18:14:42.470194 2019] [so:warn] [pid 210160] AH01574: module proxy_http_module is already loaded, skipping
[Fri Oct 18 18:14:42.470216 2019] [so:warn] [pid 210160] AH01574: module proxy_uwsgi_module is already loaded, skipping
[Fri Oct 18 18:14:42.470254 2019] [so:warn] [pid 210160] AH01574: module deflate_module is already loaded, skipping
[Fri Oct 18 18:14:42.470302 2019] [so:warn] [pid 210160] AH01574: module expires_module is already loaded, skipping
[Fri Oct 18 18:14:42.470324 2019] [so:warn] [pid 210160] AH01574: module ssl_module is already loaded, skipping
[Fri Oct 18 18:14:42.523708 2019] [alias:warn] [pid 210160] AH00671: The Alias directive in /etc/httpd/conf.d/autoindex.conf at line 21 will probably never match because it overlaps an earlier Alias.

In file: /etc/httpd/conf/httpd.conf

30:

ServerName www.genelab-galaxy.usra.edu

37:

ServerAdmin root@localhost

46:

ServerName www.genelab-galaxy.usra.edu:80

61:

Listen 80

97:

User apache

98:

Group apache

128:

# 130: Options None # 132: AllowOverride None # 135: Order allow,deny # 136: Allow from all # 141: # 142: Order allow,deny # 143: Deny from all # 144: Satisfy All # 148: # 149: Order allow,deny # 150: Allow from all # 153: # 154: Order allow,deny # 155: Options FollowSymLinks # 156: Allow from all # 160: Alias /static /configure-galaxy/static # 163: AllowEncodedSlashes NoDecode # 164: RewriteRule ^/configure-galaxy/static/style/(.*) /data/galaxy/static/june_2007_style/blue/$1 [L] # 165: RewriteRule ^/configure-galaxy/static/scripts/(.*) /data/galaxy/static/scripts/$1 [L] # 166: RewriteRule ^/configure-galaxy/static/(.*) /data/galaxy/static/$1 [L] # 167: RewriteRule ^/configure-galaxy/favicon.ico /data/galaxy/favicon.ico [L] # 168: RewriteRule ^/configure-galaxy/images/(.*) /data/galaxy/static/images/$1 [L] # 169: RewriteRule ^/configure-galaxy/robots.txt /data/galaxy/static/robots.txt [L] # 172: # 173: ProxyPass ! # 174: RequestHeader set X-URL-SCHEME http # 175: Options FollowSymLinks # 176: Allow from all # 177: Order allow,deny # 179: ExpiresActive On # 180: ExpiresDefault "access plus 6 hours" # 183: # 186: ProxyPass http://127.0.0.1:8080 # 187: ProxypassReverse http://127.0.0.1:8080 # 188: RequestHeader set X-URL-SCHEME http # 189: Options +FollowSymLinks -SymLinksIfOwnerMatch # 190: Order Allow,Deny # 191: Allow from all # 194: SetOutputFilter DEFLATE # 195: SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary # 196: SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary # 197: SetEnvIfNoCase Request_URI /history/export_archive no-gzip dont-vary # 200: Alias /favicon.ico /data/galaxy/static/favicon.ico # 201: Alias /meatball-favicon.ico /data/galaxy/static/images/meatballfavicon.ico # 202: Alias /icons /icons # 206: # 207: RewriteEngine On # 208: RewriteOptions Inherit # 209: ServerAlias genelab-galaxy.usra.edu # 210: ServerAlias wwww.genelab-galaxy.usra.edu # 214: Redirect permanent "/static" "/configure-galaxy/static" # 215: Redirect permanent "/api" "/configure-galaxy/api" # 216: Redirect permanent "/history" "/configure-galaxy/history" # 217: Redirect permanent "/login" "/configure-galaxy/login" # 218: Redirect permanent "/library" "/configure-galaxy/library" # 219: Redirect permanent "/user" "/configure-galaxy/user" # 220: Redirect permanent "/tool_runner" "/configure-galaxy/tool_runner" # 223: Redirect permanent "/galaxy" "/" # 226: Header always set Strict-Transport-Security "max-age=15552000; includeSubdomains" # 231: AddOutputFilterByType DEFLATE text/html text/plain text/xml # 232: AddOutputFilterByType DEFLATE text/css # 233: AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript # 234: AddOutputFilterByType DEFLATE application/rss+xml # 235: AddOutputFilterByType DEFLATE application/xml # 236: AddOutputFilterByType DEFLATE application/json # 243: RequestHeader set X-URL-SCHEME "%{REQUEST_SCHEME}e" # 246: ProxyTimeout 180 # 256: # 257: AllowOverride None # 258: Require all granted # 261: RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END] # 262: RewriteRule ^/plugins/(.+)/(.+)/static/(.*)$ /data/galaxy/config/plugins/$1/$2/static/$3 [L] # 263: # 264: Require all granted # 266: # 267: Require all granted # 285: DocumentRoot "/var/www/galaxy/html" # 290: # 291: AllowOverride none # 293: Require all granted # 296: # 309: Options Indexes FollowSymLinks # 315: AllowOverride none # 320: Require all granted # 328: DirectoryIndex index.html # 335: # 336: Require all denied # 345: ErrorLog "logs/error_log" # 352: LogLevel warn # 359: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined # 360: LogFormat "%h %l %u %t \"%r\" %>s %b" common # 364: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio # 380: CustomLog "logs/access_log" combined # 409: ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" # 417: # 418: AllowOverride All # 419: Options None # 420: Require all granted # 427: TypesConfig /etc/mime.types # 444: AddType application/x-compress .Z # 445: AddType application/x-gzip .gz .tgz # 466: AddType text/html .shtml # 467: AddOutputFilter INCLUDES .shtml # 476: AddDefaultCharset UTF-8 # 484: MIMEMagicFile conf/magic # 508: EnableSendfile on # In file: /etc/httpd/conf.d/autoindex.conf # 16: IndexOptions FancyIndexing HTMLTable VersionSort # 21: Alias /icons/ "/usr/share/httpd/icons/" # 23: # 24: Options Indexes MultiViews FollowSymlinks # 25: AllowOverride None # 26: Require all granted # 34: AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip # 36: AddIconByType (TXT,/icons/text.gif) text/* # 37: AddIconByType (IMG,/icons/image2.gif) image/* # 38: AddIconByType (SND,/icons/sound2.gif) audio/* # 39: AddIconByType (VID,/icons/movie.gif) video/* # 41: AddIcon /icons/binary.gif .bin .exe # 42: AddIcon /icons/binhex.gif .hqx # 43: AddIcon /icons/tar.gif .tar # 44: AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv # 45: AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip # 46: AddIcon /icons/a.gif .ps .ai .eps # 47: AddIcon /icons/layout.gif .html .shtml .htm .pdf # 48: AddIcon /icons/text.gif .txt # 49: AddIcon /icons/c.gif .c # 50: AddIcon /icons/p.gif .pl .py # 51: AddIcon /icons/f.gif .for # 52: AddIcon /icons/dvi.gif .dvi # 53: AddIcon /icons/uuencoded.gif .uu # 54: AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl # 55: AddIcon /icons/tex.gif .tex # 56: AddIcon /icons/bomb.gif /core # 57: AddIcon /icons/bomb.gif */core.* # 59: AddIcon /icons/back.gif .. # 60: AddIcon /icons/hand.right.gif README # 61: AddIcon /icons/folder.gif ^^DIRECTORY^^ # 62: AddIcon /icons/blank.gif ^^BLANKICON^^ # 68: DefaultIcon /icons/unknown.gif # 86: ReadmeName README.html # 87: HeaderName HEADER.html # 93: IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t # In file: /etc/httpd/conf.d/shib.conf # 20: ShibCompatValidUser Off # 25: # 26: AuthType None # 27: Require all granted # 34: # 35: AuthType None # 36: Require all granted # 38: Alias /shibboleth-sp/main.css /usr/share/shibboleth/main.css # 49: # 50: AuthType shibboleth # 51: ShibRequestSetting requireSession 1 # 52: require shib-session # In file: /etc/httpd/conf.d/ssl.conf # 5: Listen 443 https # 18: SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog # 23: SSLSessionCache shmcb:/run/httpd/sslcache(512000) # 24: SSLSessionCacheTimeout 300 # 36: SSLRandomSeed startup file:/dev/urandom 256 # 37: SSLRandomSeed connect builtin # 49: SSLCryptoDevice builtin # 56: # 64: ErrorLog logs/ssl_error_log # 65: TransferLog logs/ssl_access_log # 66: LogLevel warn # 70: SSLEngine on # 75: SSLProtocol all -SSLv2 -SSLv3 # 80: SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA # 100: SSLCertificateFile /etc/pki/tls/certs/localhost.crt # 107: SSLCertificateKeyFile /etc/pki/tls/private/localhost.key # 175: # 176: SSLOptions +StdEnvVars # 178: # 179: SSLOptions +StdEnvVars # 208: BrowserMatch "MSIE [2-5]" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0 # 214: CustomLog logs/ssl_request_log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" # In file: /etc/httpd/conf.d/userdir.conf # 17: UserDir disabled # 31: # 32: AllowOverride FileInfo AuthConfig Limit Indexes # 33: Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec # 34: Require method GET POST OPTIONS # In file: /etc/httpd/conf.d/welcome.conf # 8: # 9: Options -Indexes # 10: ErrorDocument 403 /.noindex.html # 13: # 14: AllowOverride None # 15: Require all granted # 18: Alias /.noindex.html /usr/share/httpd/noindex/index.html # 19: Alias /noindex/css/bootstrap.min.css /usr/share/httpd/noindex/css/bootstrap.min.css # 20: Alias /noindex/css/open-sans.css /usr/share/httpd/noindex/css/open-sans.css # 21: Alias /images/apache_pb.gif /usr/share/httpd/noindex/images/apache_pb.gif # 22: Alias /images/poweredby.png /usr/share/httpd/noindex/images/poweredby.png # In file: /etc/httpd/conf.d/wsgi-myproxy-oauth.conf # 3: Alias /oauth/templates/ /usr/share/myproxy-oauth/myproxyoauth/templates/ # 4: # 5: Options Indexes # 6: Require all granted # 10: Alias /oauth/static/ /usr/share/myproxy-oauth/myproxyoauth/static/ # 11: # 12: Options Indexes # 13: Require all granted # 17: WSGIDaemonProcess myproxyoauth user=myproxyoauth group=myproxyoauth threads=1 # 18: WSGISocketPrefix run/wsgi # 20: WSGIProcessGroup myproxyoauth # 21: WSGIScriptAlias /oauth /usr/share/myproxy-oauth/wsgi.py # 22: # 25: SSLRequireSSL # 26: Require all granted # 28: # 31: SSLRequireSSL # 32: Require all granted httpd (pid 208748) already running [kchawla@galaxy ~]$

Nope, I am not running any .htaccess

27

if its a bug then can i use another version of certbot to get the cert ?

Thanks !!

28

Can you try (temporarily) disabling?:
194: SetOutputFilter DEFLATE
231: AddOutputFilterByType DEFLATE text/html text/plain text/xml

29

Did that but no luck

30

There may be a permanent fix for this specific situationā€¦
Step one: Replace Apache with NGINX.
[LOL]

31

Canā€™t, itā€™s a production server !!

32

The way that people have been able to work around this bug in the past is usually by -a webroot -i apache instead of --apache. In that case, you need to specify a directory from which your web server serves static content (usually your Apache DocumentRoot), and Certbot will place the challenge files into that directory instead of creating its own tempoary VirtualHost.

We still hope to figure out why this problem is affecting people and what we could do to fix it!

33

Can you add NGINX (in addition to Apache) ?
Do you serve any content with port 80 (or just 443) ?

34

Do you mind sending me the full command and I can try.

Thanks !!

35

You can run

sudo certbot -a webroot -i apache -d genelab-galaxy.usra.edu

It will ask you for the webroot path (you can also specify that on the command line with -w).

1 Like
36

...but you also mention "production"..
Are you the only admin to this server?
Is it possible someone else added an .htaccess file?
find / -name '.htaccess'

also:
DocumentRoot "/var/www/galaxy/html"

And...
[I haven't given up on adding NGINX.]
If you only use port 80 to redirect to 443, you can delegate 80 to NGINX
Then it can handle the cert renewals and redirect all else to 443

39

Hi, I tried that but no luck:

sudo certbot -a webroot -i apache -w /var/www/galaxy/html/ -d genelab-galaxy.usra.edu
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer apache
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for genelab-galaxy.usra.edu
Using the webroot path /var/www/galaxy/html for all unmatched domains.
Waiting for verificationā€¦
Challenge failed for domain genelab-galaxy.usra.edu
http-01 challenge for genelab-galaxy.usra.edu
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

40

even this command doesnā€™t work

sudo certbot -a webroot -i apache -d genelab-galaxy.usra.edu
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer apache
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for genelab-galaxy.usra.edu
Input the webroot for genelab-galaxy.usra.edu: (Enter ā€˜cā€™ to cancel): -w /var/www/galaxy/html

-w /var/www/galaxy/html does not exist or is not a directory

Input the webroot for genelab-galaxy.usra.edu: (Enter ā€˜cā€™ to cancel):

/var/www/galaxy/html is my web directory

41

At this input you don't need the "-w"

At this command you can add the webroot:
sudo certbot -a webroot -i apache -d genelab-galaxy.usra.edu

as:

sudo certbot -a webroot -i apache -d genelab-galaxy.usra.edu -w /var/www/galaxy/html
42

Do you have something in your web server configuration that would forbid serving files and directories beginning with a dot?