Migrating WordPress with HTTPs Secured by LetsEncrypt to New Server

Server
1

I have spent countless hours trying to migrate my wordpress website to a new server (VPS) but while this is not a big deal under http, I find it an extremely laborious and error prone process with Let's Encrypt. I am probably missing something so I would like to ask what I'm doing wrong or misunderstanding.

Here is my story.

To start with, both the old and the new server use NGINX as their webserver. Let's say my site is at mydomain.com as well as www.mydomain.com on the old server. Since I wasn't sure how long the migration would take me and whether it would work fine, I created a new subdomain testwp.mydomain.com and pointed it to the new server. The intention was to setup things under that domain and once it's working, point mydomain.com and www.mydomain.com at the new server.

However, while that works fine without SSL, things get complicated when I want to get a Let's Encrypt certificate for the new server because I can only do that once the DNS (with the production URI) is actually pointed to it. So last night, I temporarily pointed the DNS to the new server in order to create certificates for mydomain.com and www.mydomain.com (as well as testwp.mydomain.com) on the new server. Once that was done, I pointed the DNS back to the old site so that I could work without time pressure on the new site. For some reason, I was unable to access the new server via https://testwp.mydomain.com so I changed the NGINX configuration back to http (which is a laborious process in itself, if you are new to webservers in general and NGINX in particular).

But once I was able to access http://testwp.mydomain.com (without SSL) again, I managed to migrate the Wordpress site and I was able to use it just fine under that URL.

So the next step was to activate SSL and point the DNS to the new server for good. So I changed the DNS entry and edited the /etc/nginx/sites-enabled/default so that all http (port 80) requests are forwarded to https (port 443). (Before I made those changes, I made a small edit to my Wordpress front page so that I would be able to see, when https://www.mydomain.com gives me the new website rather than the old.)

Unfortunately, even after an hour, I am still seeing the old website under https://mydomain.com (I might add that the TTL for all my DNS entries is 1h). www.mydomain.com is for inexplicable reasons forwarded to testwp.mydomain.com (the URL in the browser's address bar changes) and I get an error NET::ERR_CERT_COMMON_NAME_INVALID. In a way, this is not surprising because, as mentioned above, the testwp.mydomain.com certificate never worked (I probably simply forgot to include it in the letsencrypt command the last time I ran it), but I don't understand why it goes to testwp.mydomain.com when I call www.mydomain.com...

More specifically it tells me

"This server could not prove that it is testwp.mydomain.com; its security certificate is from mydomain.com."

So this kind of lets me hope that once my ISP's DNS resolves mydomain.com as the new server IP at least https://mydomain.com will work (since the certificate is there), but it is still puzzling why this is taking so long and why www.mydomain.com keeps getting redirected to testwp.mydomain.com. Update: Also mydomain.com now gets redirected to testwp.mydomain.com. Could this be due to a setting in wordpress (I think the site URL in wordpress might still be set to testwp...)

Needless to say that in all that mess, it doesn't really help that there is a delay between changing the DNS server and your browser actually using the new IP. Plus, as I had to learn, most browsers will not accept to connect with a site via http if they once connected to that same site via https. In order to make them use http again, I had to dig deep into the browser config and make it forget that my domain once used http ( return 301 https://$server_name$request_uri;) but

What I'm mainly wondering about is if there is an easier way of switching back and forth between http and https without having to completely change your /etc/nginx/sites-enabled/default every time. But any explanations and hints with regard to my above story would be welcome.

BTW: if you are wondering what I did not copy my existing certificates from the old server to the new server, the reason is that I simply don't know how to do it and trying (as I often do) would have opened yet another can of worms which would have been just too much on top of all the rest.

2

In general, I think the easiest way to move a site to a different server is to first request a new certificate on the new server using certbot's --manual plugin. This allows you to validate the domain by manually creating a file on the site, which you can do on the old server while the DNS is still pointing at it, even though certbot itself is running on the new server. Then you can test using your hosts file and make sure everything is working properly before you make any changes to the DNS. Once you have everything migrated and the DNS pointed at the new server, remember to get a new certificate with --webroot or another plugin (or directly modify the configuration) so that renewals will work correctly.

I'm not sure if it's too late for you to do this; your site (if indeed it is the one you forgot to redact in one place in your post) seems to be working fine at the moment. If that's because the DNS is still pointing at your old server, you can probably still try the above. If it's because you managed to fix it on your own... great!

This may be because you've enabled HTTP Strict Transport Security (also known as HSTS). If you want to disable this while you're testing things, look for strict-transport-security in your configuration and temporarily set the max-age to 0.

1 Like
3

I will have a look at that asap (it's this you mean, right?)

I'm not sure what the host file has to do with this or how I'd use it to test something. I have never even looked at /etc/hosts/ in relation to SSL... And in the case of my VPS hoster, the IPs in the hostfile are not even the public IPs of my machine...)

Could you explain this a bit more? Why do I need to get a new certificate right away? Currently, I have simply set up a cron job for letsencrypt renew as described in this tutorial.

Yes, it's because of HSTS, but because the strict-transport-security setting was not part of my configuration and I didn't know where to add it, I chose another workaround: I made my browser forget what it knows about my domain.

4

Right (note that certbot and letsencrypt are just different names for the same program).

I mean the hosts file on your personal computer, not the one on the server. Enter the IP address of the new server alongside the domain name you want to test, in the hosts file on your computer. This will effectively override the DNS for that domain, but only when viewed on your computer. So you can point your web browser at the domain and see the site on the new server, before the new DNS settings are live.

Because, if you did as I suggested, the fact that you used --manual will be remembered and it will try to do the same thing again when the cron job runs - but this will fail because you won't be around at the terminal to read and follow the instructions. So you need to update the renewal configuration, and the simplest way to do that is to request a new certificate using the proper plugin. You can also edit the configuration file manually, if you prefer.

1 Like
5

Oh great, now I just installed certbot too (in addition to letsencrypt). Is that a problem?

6

It’s generally not a problem to have both installed, but if you actually run the newer version (which is almost certainly certbot, as that’s the newer name) it could update the configuration files to a format that may be incompatible with the older version. So if you already did that you’ll have to update your cron job to use certbot too.

8

I truly appreciate your explanation. That was important. So I now got the certificate via the dns challenge using letsencrypt certonly -d sub.domain.com --manual --preferred-challenges dns and instead of partying all night because it worked, I then re-ran the whole thing using letsencrypt certonly -a webroot --webroot-path=/var/www/html -d sub.domain.com So since I already have my cronjob for letsencrypt setup, am I right in assuming that my certificates will be automatically renewed, including the one originally obtained manually via dns?

9

Yes, letsencrypt certonly overwrites the renewal configuration as long as you use the exact same set of domains (which you did), so your existing cron job should work.

I take it that means you’ve got everything working now?

10

I agree with you and I tend to plan things a lot for that very reason. Buy in this case, there was so much to plan and read up on that let's encrypt appeared arguably a minor element in the process. I did not mention the whole complexity in the OP because it was complicated enough to describe: What I left out is that I not only migrated a WordPress site but also a Discourse forum running in a Docker container on the same server and mapped onto its own subdomain and using the NGINX server as a reverse proxy. So I studied exactly how to do this and did not pay too much attention to SSL because when I originally set up the first server, setting up SSL using letsencrypt was so easy, so why should I worry doing it a second time?

And I did almost exactly as you describe and created a dry-run subdomain (except I called it "testwp" (and "test" for the discourse forum) but my conclusion after all this odyssey is that this was precisely the mistake: I should just have accepted some down-time of the site and skip the dry-run setup. It would have been so easy.

For anyone who is not an expert at neither let's encrypt nor webserver administration (like myself) I would recomment this as one of two options. The other option is obtaining the certificates using the dns challenge during the dry-run phase like this:

A third option is manual mode without the dns challenge but I just couldn't get that one to work for the forum subdomain And seeing how easy the dns challenge is, I see no reason to ever again use manual without dns. The only tricky part with the dns challenge is that you apparently have to wait a bit after deploying your token to the dns server and hitting enter to make letsencrypt find it. The first time it didn't work but the second time I waited a couple of minutes (just to be sure) and tada :tada: it worked.

1 Like
11

Uhm, no, actually I didn't :frowning: (I simplified the story a bit) This is what I did:

First: letsencrypt certonly -d forum.mydomain.com --manual --preferred-challenges dns

And then (once the DNS pointed to the new server): sudo letsencrypt certonly -a webroot --webroot-path=/var/www/html -d mydomain.net -d www.mydomain.net -d forum.mydomain.net -d test.mydomain.net

(Reason: I already had the other certs on the new server so there was no need of including them in the manual DNS run.)

So this is not going to work?

12

It will work, in that it should be able to successfully renew the new, combined certificate that you obtained covering all four domains. It will also try, and fail, to renew the separate certificate that you obtained just for forum.mydomain.com. As long as nginx isn’t configured to use that certificate, it shouldn’t be a problem. You can prevent the renewal attempt by deleting the corresponding file from /etc/letsencrypt/renewal. Note that you’ll still get reminder emails from Let’s Encrypt when that certificate is nearing expiry; you can safely ignore those as long as the other certificate is renewed correctly.

You should double-check that nginx is indeed using the combined certificate for that subdomain, though.

13

I guess I don't have such a combined certificate:

$ sudo letsencrypt certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Found the following certs:
  Certificate Name: mydomain.net
	Domains: mydomain.net test.mydomain.net www.mydomain.net
	Expiry Date: 2017-07-01 04:01:00+00:00 (VALID: 88 days)
	Certificate Path: /etc/letsencrypt/live/mydomain.net/fullchain.pem
	Private Key Path: /etc/letsencrypt/live/mydomain.net/privkey.pem
  Certificate Name: test.mydomain.net
	Domains: testwp.mydomain.net test.mydomain.net
	Expiry Date: 2017-07-01 00:54:00+00:00 (VALID: 88 days)
	Certificate Path: /etc/letsencrypt/live/test.mydomain.net/fullchain.pem
	Private Key Path: /etc/letsencrypt/live/test.mydomain.net/privkey.pem
  Certificate Name: forum.mydomain.net
	Domains: forum.mydomain.net
	Expiry Date: 2017-07-02 20:57:00+00:00 (VALID: 89 days)
	Certificate Path: /etc/letsencrypt/live/forum.mydomain.net/fullchain.pem
	Private Key Path: /etc/letsencrypt/live/forum.mydomain.net/privkey.pem

Well, at least I've got 88 days to solve this problem...

14

That’s odd; the command you posted above, sudo letsencrypt certonly -a webroot --webroot-path=/var/www/html -d mydomain.net -d www.mydomain.net -d forum.mydomain.net -d test.mydomain.net, would have attempted to obtain such a certificate. Are you sure it succeeded?

15

Since I still have the terminal open, I can share the output:

$ sudo letsencrypt certonly -a webroot --webroot-path=/var/www/html -d mydomain.net -d www.mydomain.net  -d forum.mydomain.net -d test.mydomain.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org

-------------------------------------------------------------------------------
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/mydomain.net.conf)

It contains these names: mydomain.net, test.mydomain.net, www.mydomain.net

You requested these names for the new certificate: mydomain.net, www.mydomain.net,
forum.mydomain.net, test.mydomain.net.

Do you want to expand and replace this existing certificate with the new
certificate?
-------------------------------------------------------------------------------
(E)xpand/(C)ancel: e
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain.net
http-01 challenge for www.mydomain.net
http-01 challenge for forum.mydomain.net
http-01 challenge for test.mydomain.net
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/html/.well-known/acme-challenge
Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/mydomain.net/fullchain.pem. Your cert will
   expire on 2017-07-02. To obtain a new or tweaked version of this
   certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le
16

I noted in another thread that, although the letsencrypt program has been renamed to certbot, packages for certbot still offer letsencrypt as an alias; in this case, if both came from the same package on a given system, they'll both work in the same way and running either won't create incompatibilities for the other. We do recommend certbot exclusively for all documentation to ensure that people are using and becoming familiar with the new name.

17

Aha! The devil is in the details: my letsencrypt and certbot come from entirely different sources.letsencrypt was installed via apt-get install letsencrypt if I remember correctly and certbot was much less straight forward:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot

The OS is Ubuntu 16.04.

So they don't work side-by-side?

18

True, then those should not be used interchangeably. I suggest only using whichever one is newer (probably certbot), after confirming with --version.

19

That was close:

P.S. I think 10 characters minimum requirement for posts is enough (the default setting in discourse)

20

So now we still have the mystery of why certbot (or letsencrypt) reported that it had successfully obtained a certificate for all four domains, but that certificate does not seem to exist on the server. Are you sure you ran both commands (sudo letsencrypt certonly ... and sudo letsencrypt certificates) on the same server?

Did you remove the line from /etc/hosts after testing?

21

Yes. I have it all in front of me in one terminal window.

I never did anything with /etc/hosts. What line are you referring to?

Meanwhile, I can confirm that the mydomain.net certificate (the first one in the list above) does indeed not seem to contain the forum subdomain because I just noticed that when I call forum.mydomain.net it gives me a NET::ERR_CERT_COMMON_NAME_INVALID, stating that the mydomain.net is not valid for this domain, i.e. for forum.mydomain.net.

So I had to change my NGINX configuration so that the forum.mydomain.net certificate (the third one in the list above) is used in one server block and the mydomain.net certificate in another.

I'm starting to wonder whether we are perhaps looking at a bug...

Update: see here (this is driving me insane) :dizzy_face: