Can't use wget to download https letsencrypt sites on server, but on Mac it works. Doesn't work for Facebook scraper


#1

I have https via LetsEncrypt installed on my domain cantpostlinks://blog.andytriboletti.com
No errors in the browser on Mac or on iPhone.
I noticed a problem where Facebook’s open graph URL scraper used for previews wasn’t working.

After investigating I noticed that wget cantpostlinks://blog.andytriboletti.com works on my Macbook, but does not work on my Centos server. On Mac I am using wget version GNU Wget 1.17.1 built on darwin15.2.0. On Centos I am using GNU Wget 1.14 built on linux-gnu. I upgraded to GNU Wget 1.18 built on linux-gnu and it still doesn’t work.

How Debugger sees your URL is blank:
cantpostlinks://developers.facebook.com/tools/debug/echo/?q=http%3A%2F%2Fblog.andytriboletti.com%2F

I have opened a bug with Facebook: cantpostlinks://developers.facebook.com/bugs/546742535513914/

I have tried posting site with Google Plus and the preview works: cantpostlinks://plus.google.com/+AndyTriboletti/posts/3uzLN83Ysgn

Full output of wget --version and it working to download cantpostlinks://blog.andytriboletti.com/ on my Macbook:

Andys-MBP:~ andytriboletti$ brew upgrade wget
Error: wget 1.17.1 already installed
Andys-MBP:~ andytriboletti$ wget --version
GNU Wget 1.17.1 built on darwin15.2.0.

+digest -gpgme +cantpostlinks +ipv6 -iri +large-file -metalink -nls +ntlm 
+opie -psl +ssl/openssl 

Wgetrc: 
    /usr/local/etc/wgetrc (system)
Compile: 
    clang -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc" 
    -DLOCALEDIR="/usr/local/Cellar/wget/1.17.1/share/locale" -I. 
    -I../lib -I../lib -I/usr/local/opt/openssl/include -DNDEBUG 
Link: 
    clang -DNDEBUG -L/usr/local/opt/openssl/lib -lssl -lcrypto -ldl -lz 
    ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a 

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.
Andys-MBP:~ andytriboletti$ wget cantpostlinks://blog.andytriboletti.com/
--2016-06-20 15:44:41--  cantpostlinks://blog.andytriboletti.com/
Resolving blog.andytriboletti.com... 104.237.138.56, 2600:3c00::f03c:91ff:fe84:6879
Connecting to blog.andytriboletti.com|104.237.138.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html.16'

index.html.16                         [ <=>                                                       ]  55.68K  --.-KB/s    in 0.09s   

2016-06-20 15:44:41 (641 KB/s) - 'index.html.16' saved [57012]

When I run this on my Linux server I do get an error:

-bash-4.2$ wget cantpostlinks://blog.andytriboletti.com/
--2016-06-19 13:10:44--  cantpostlinks://blog.andytriboletti.com/
Resolving blog.andytriboletti.com (blog.andytriboletti.com)... 2600:3c00::f03c:91ff:fe84:6879, 104.237.138.56
Connecting to blog.andytriboletti.com (blog.andytriboletti.com)|2600:3c00::f03c:91ff:fe84:6879|:443... connected.
ERROR: no certificate subject alternative name matches
	requested host name ‘blog.andytriboletti.com’.
To connect to blog.andytriboletti.com insecurely, use `--no-check-certificate'.

As well as wget, it’s also an issue with curl.

Facebook URL previews work with a domain I bought using a Namecheap cert.

Originally posted here: cantpostlinks://stackoverflow.com/questions/37946111/cant-use-wget-to-download-cantpostlinks-letsencrypt-sites-on-server-but-on-mac-it-work


#2

Your domain resolves to an IPv6 address that does not seem to be serving the correct certificate. Here are two wget tests, one requesting your site via IPv4 and one via IPv6:

user@host:~# wget -4 https://blog.andytriboletti.com
--2016-06-21 15:58:16--  https://blog.andytriboletti.com/
Resolving blog.andytriboletti.com (blog.andytriboletti.com)... 104.237.138.56
Connecting to blog.andytriboletti.com (blog.andytriboletti.com)|104.237.138.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                      [  <=>                                                                                      ]  55.68K   249KB/s    in 0.2s

2016-06-21 15:58:18 (249 KB/s) - ‘index.html’ saved [57012]

user@host:~# wget -6 https://blog.andytriboletti.com
--2016-06-21 15:58:22--  https://blog.andytriboletti.com/
Resolving blog.andytriboletti.com (blog.andytriboletti.com)... 2600:3c00::f03c:91ff:fe84:6879
Connecting to blog.andytriboletti.com (blog.andytriboletti.com)|2600:3c00::f03c:91ff:fe84:6879|:443... connected.
ERROR: no certificate subject alternative name matches
	requested host name ‘blog.andytriboletti.com’.
To connect to blog.andytriboletti.com insecurely, use `--no-check-certificate'.

This would cause issues for users or systems that support IPv6, while everything would look fine with IPv4. Make sure that the IPv6 address is correct and that your server supports IPv6 (and that it’s enabled in your web server), or remove the AAAA record for that domain.

Small aside: SSL Labs shows a number of issues with your SSL configuration.
Take a look at the Mozilla SSL Configuration Generator for a good starting point for a secure config.


#3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.