[Solved] Apple server proxy blocks access to .well-known

ctgreybeard · 2015-11-30 03:53:59 UTC

I’ve run into a problem and I am hoping someone here has a suggestion.

I run an Apple Server on the network and I wanted to try LE on a backend Jessie (Raspberry Pi 2).

I set the Apple server VirtualHost as a reverse proxy and I can access the RPi fine from the internet using the Pi’s external host name.

EXCEPT, that the built-in proxy that actually listens on the external port 80 and 443 is capturing /.well-known with:

ProxyPass /.well-known unix:/var/run/caldavd_requests/unsecured.sock|http://localhost/.well-known
ProxyPassReverse /.well-known unix:/var/run/caldavd_requests/unsecured.sock|http://localhost/.well-known
SetEnvIf Request_URI "/.well-known.*" proxy-nokeepalive=1

Unfortunately that is in the config that is buried within the Server.app and I cannot change it. I CAN, however, change the conf that Includes that file.

What’s the best way to handle this? Should I complain to Apple? The proxy is set up for CalDAV and the like. I don’t use those services but I have little control over them or their configuration.

Any thoughts would be appreciated.

Bill W

ctgreybeard · 2015-11-30 04:17:50 UTC

I have “solved” this at least temporarily but I’m not sure it’s the “right” thing to do.

I added:

ProxyPass "/.well-known/acme-challenge" "!"

ahead of any other ProxyPass requests for that directory (there are several in the various configs) and it now passes the request on to the RPi.

Is this a proper fix?

Bill W

ctgreybeard · 2015-11-30 23:36:47 UTC

Well, it turns out I was completely overthinking the problem!

Apple, in their wisdom, runs a two-level web service. The first level just proxies to other services and local web sites. I was trying to put my proxy info into the second-level service. All I needed to do was put the proxy to my backend server at the first level proxy. It is in one of the proxy definitions at that level that blocks (actually just re-routes it) access to /.well-known. As a separate virtual-host my backend server gets to see everything.

johntucker · 2015-12-19 06:36:11 UTC

ctgreybeard,

I don’t understand what your problem. I think I have a similar problem in that I can’t get the Mac OS Server to serve up a file that starts with a dot (ex. .well-known/…) if I remove the dot the file in the acme-challenge gets served but it is not found if the dot is present. If you know how to get around the dot problem I have complete control of the server configuration but I don’t know where to start.

Thanks.

ctgreybeard · 2015-12-21 01:54:30 UTC

You will not be able to use .well-known on a Mac OS X Server. I was able to use that on another system that the Server is a reverse-proxy for but the internals of the setup for a OS X Server route .well-known to internal processes. It would take major surgery to get around that.

Bill

cyrilpic · 2015-12-24 11:08:12 UTC

In the latest Server app (5.0.15) there are two instances of Apache running one in front proxying the request either to a service (Calendar, Wiki, Xcode, etc.) or the the 2nd Apache servicing websites.

.well-known is caught by the Calendar service (independently of its status).

One solution I have found (and it is not even so dirty) is to put a file like this one :

gist.github.com

https://gist.github.com/cyrilpic/eea4c66246f1829c804f

apache_serviceproxy_customsites_disableproxy.conf

<VirtualHost *:80>
   ServerName www.example.com
   ProxyPreserveHost On
   SetEnv proxy-chain-auth on
   RequestHeader set X-Forwarded-Proto "http"
   RequestHeader set X-Forwarded-Port "80"
   ProxyPass / http://127.0.0.1:34580/
   ProxyPassReverse / http://127.0.0.1:34580/
</VirtualHost>

into /Library/Server/Web/Config/Proxy and the restart the web service.

DDJarod · 2015-12-30 16:28:47 UTC

The cleanest solution I found was to create a new custom proxy site for apples apache serviceproxy:

/Library/Server/Web/Config/Proxy/apache_serviceproxy_customsites_letsencrypt.conf:
ProxyPass /.well-known/acme-challenge http://127.0.0.1:34543/.well-known/acme-challenge ProxyPassReverse /.well-known/acme-challenge http://127.0.0.1:34543/.well-known/acme-challenge

That file is automatically loaded by the proxy thru the file /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf

After I added the file I restarted the proxy thru launchctl:
sudo launchctl stop com.apple.serviceproxy because the proxy itself doesn’t restart when the web service is restarted.

I execute letsencrypt from the github checkout as follows:
./letsencrypt-auto certonly --webroot -w /Library/Server/Web/Data/Sites/Default -d my.domain.com

majorsl · 2016-01-01 17:35:28 UTC

Thank you for this, it was just what I was looking for.

MajorCGN · 2016-03-09 18:19:01 UTC

Sorry, both solutions did not work for me under 10.11.3 El Capitan and Server 5.0.15.

The double proxy is such a non-intuitive solution just to get that idiotic “apple default server”. Linux is easier that this crappy ****

That idea to use a dotted directory seems to be really a no-go for OSX and letscrypt. The goal to do easy encryption ist totally foiled under OSX server. I need about 3 minutes to get a certificate the “classical way” via CSR.

I didn’t managed to get that dotted directory to get served in 90 minutes just to verify that it is “my server”. Everything else works - python, pip, virtualenv - no errors, like a charm.
So it is 30 times easier to get a cert via the “old” way with CSRs. This is “Letsforgetencryption” for OSX server and a nerdy thing again, not easy, not better than classical ways.
No - it could not get it to work like the 3 minute certs - so ist more than 30 times harder to use.

Okay - i understand i have to break the interception of dotted directory names by Calendar service for other services with a new proxy to serve it to the public - namely LE, which must use that **** dotted path. Without a dot it is not my server. For sure.

Tried that solution of cyrilpic. Stopped proxy, restartet webservice. cleared browser caches. No difference.
When trying to access .well-known directly i am redirected to a CalDAV-Service. Going deeper to /acme-challenge/ or (file) i get “not found”.

Tried that not so well described solution of DDJarod. Did not work because /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf would not load */Library/Server/Web/Config/Proxy/apache_serviceproxy_customsites_letsencrypt.conf
The server was not reachable at all, not by me and of course not by LE.
Reverted and got it running again, but with no access to dotted directories.

Gave up. 90 minutes just to get around a dot is much money. i can spend in a commercial cert.

This is really a little stupid idea, that little dot. A not dotted directory would be accessible with no problems. Google uses files on the root directory for a similar task. Works. Because they do not use DOTS.
Dont know, who had that bad idea which excludes OSX servers with no feasible reason. One dot less, and there would be less problems for OSX.

Nit · 2016-03-09 23:00:34 UTC

The internet standards https://tools.ietf.org/html/rfc5785. URI under .well-known must be registered and so are sure to no collide with another application.

Osiris · 2016-03-09 23:16:25 UTC

Furthermore, don’t blame Let’s Encrypt for something what Apple broke.

By the way, with Let’s Encrypt, you also can use CSRs. CSRs don’t have anything to do with how CA’s validate the claim of ownership for a domain.

MajorCGN · 2016-03-10 08:16:18 UTC

Yep.

But for example google does not use this standard and works better by securing the ownerschip of a domain.
The owner has to put a single file into the root directory and that’s it.
Similar, but much simpler and i don’s see huge security reasons for a dotted directory.

Standards , especially “Request For Comments”, are not a physical law and there would be less hassle without .well-known

I agree, Apple violates this standard and one should blame Apple for this.
But that does not help LE in any way.

MajorCGN · 2016-03-10 08:24:23 UTC

As i said, it is about 30 times easier to get a cert with the workflow of Apples certificate app and now tell me, why LE ?

Not easier - five clicks in OSX server vs. installing unsure packets from unknown sources in hours of BASHing.
Not cheaper - hassle is money
Not more secure - same shit as the normale certs

As an investor, i wouldn’t put a single penny in this project, because it is not the promised breakthrough in any way.

cyrilpic · 2016-03-10 08:50:11 UTC

Well, nobody forces you to use LE. If you are not happy, don’t use it.

Concerning the dot-files, these tricks do work (mine should only work for “custom” domains). So, there might be something not standard in your configuration. If you want help configuring this, you might want to give us more information about your setup (e.g. configuration files).
You might also want to wait for the upcoming Server 5.1 update (along with El Capitan 10.11.4), as part of the dot-file bug has been corrected (Apple acknowledged it was a bug). Meaning, service from Apple (wiki, calendar, xcode) will not be served any more for custom domains (i.e the file I suggested in this conversation will be generated automatically by the Server App).

Osiris · 2016-03-10 08:50:22 UTC

Just because it doesn’t work for your specific situation, doesn’t mean it won’t work for 99,999% of all other users…

MajorCGN · 2016-03-10 09:09:08 UTC

A simple google search shows: it does not work for 99.999% of Apple server users, because it’s a bug, which prevents the usage.

And: 99.999% of servers are not Apple (and most of them know why).

MajorCGN · 2016-03-10 09:10:24 UTC

Thank you for that answer ! Did not know that.

Maybe i’ll try later with Server 5.1

Osiris · 2016-03-10 09:10:55 UTC

Yes indeed, for a very small number of users there’s a bug in Apple’s software. So Let’s Encrypt still is the revolution in TLS everyone is waiting for

You can’t expect Let’s Encrypt to ditch their original ACME-specification just because one vendor has a buggy piece of software, now can you?