According to the HSTS listed domain names rules, .dev websites are required to be served over HTTPS
My domain is: mysite.dev (this is not really it, I don't want to post it here yet)
I ran this command:
certbot certonly --manual -d mysite.dev -d *.mysite.dev
It produced this output: I successfully passed the DNS-01 challenge. Then, it asked me to upload a file to my website root for the HTTP-01 challenge.
And there is the problem. I can't load it into my root directory and access it via http:// since all files have to be served over https to begin with. (It's catch 22)
My web server is (include version): Ubuntu 20.04
The operating system my web server runs on is (include version): linux
My hosting provider, if applicable, is:
I can login to a root shell on my machine (yes or no, or I don't know):
I cannot log into a shell for this account. However, I can upload generated CRT files through cPanel
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): cPanel
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.40.0-1ubuntu0.1
I'm not familiar with the restriction on .dev, but does the server not have port 80 open at all? Even if usual user web browsers won't, I'd expect that Let's Encrypt servers would work on it.
In any event, if you're using DNS-01 for the wildcard name, can't you just use DNS-01 for the non-wildcard name as well? Add --preferred-challenges dns to your certbot command.
(Though what you really want to do is automate the whole process, including the DNS challenge and installing the certificate. Being able to do it manually can be nice to get started, but doing it every couple months and your site being inaccessible if you forget can be a pain.)
Google has opted to have their TLDs (like .dev) included in the HSTS Preload List. This does not preclude websites from having port 80 open or serving content over HTTP. It simply means that browsers will initially make requests over port 443 (HTTPS). Let's Encrypt is not a browser and always initially makes its http-01 challenge file requests over port 80 (HTTP). These requests can be forwarded to port 443 (HTTPS), but not any other port.
HSTS is only a browser restriction. Every program code / curl / online tool can check your port 80.
See the Grade A+ - domains from "check-your-website". All with a working port 80, a redirect http -> https and preloaded (HSTS + hardcoded "use always https").
Ok, adding --preferred-challenges=dns solved the issue of removing http-01 challenge.
I think I found a bug though in how certbot issues the DNS challenge instructions. When running it, it asks me to input the TXT record as _acme-challenge.mydomain.dev with the corresponding value to type in.
Letsdebug shows a "duplicate DNS" issues and asks me to add a period at the end of the .dev. name, or remove mydomain.dev completely from the TXT record title. The first option didn't work, however the second did, so I ended up with a TXT record of _acme-challenge.
Once I did that, everything went through ok. Should I report that somewhere?
That's an idiosynchrasy of your DNS hosting software. Some providers require only the portion of the name that is not part of the zone, others let you put the fully qualified name which includes the domain. The instructions provided by the client try to avoid ambiguity by always giving you the full FQDN that will be queried by the challenge validators. But it's up to you to interpret that and plug it into your DNS provider properly (or ideally just let a DNS plugin handle it for you).
