Help us test renewal with "letsencrypt renew"

Client dev
1

Hi folks,

We know a lot of Public Beta users will need to renew certificates soon and our existing recommendations for how to do so require a little more scripting than we’d like. (Fortunately, people have posted a lot of recipes that are working well for them in various environments—thanks for those contributions!)

We wanted to offer something better in our Python client to automatically renew certs that are near expiry (for people who got their certs using that client). We’ve decided on a new approach called letsencrypt renew, which does a better job of remembering the settings you chose when obtaining a cert, and of reporting the results of renewal cleanly.

We’d like for people who are willing to give this a try and let us know how it works for you!

To help us test letsencrypt renew before the 0.4.0 release, you need to get the development version of the client from git:

https://letsencrypt.readthedocs.org/en/latest/contributing.html#running-a-local-copy-of-the-client

Then you can run it and tell us what happens and whether you found anything confusing or whether anything was broken in your situation. Currently, letsencrypt renew tries to renew all certificates that it regards as due for renewal. If you have a large number of certificates, you may want to consider how this interacts with the rate limit; there’s another thread talking about our efforts to make sure that the rate limit won’t interfere with certificate renewals.

Q. What’s the difference between letsencrypt renew and letsencrypt certonly?
A. letsencrypt certonly is used to obtain a single certificate (which might be a renewal of a previous one if you specify exactly the same domain names with -d). It aims to let you choose parameters for that cert from scratch, rather than using any of the parameters from a previously-obtained cert. letsencrypt renew is used to renew many certificates (if necessary), using the same parameters and defaults that were originally chosen at the time that each one was first obtained.

Q. How does letsencrypt renew remember my settings?
A. letsencrypt renew uses a set of conf files in /etc/letsencrypt/renew to remember how certificates were obtained. For now, it’s advisable to view those files as a magical black box; in version 0.5.0 we’ll be cleaning them up and documenting their contents for users who want to tweak things. Important note: if you generated a certificate with client versions 0.3.0 or below, and then renewed it manually using different settings, you may need to perform a another manual renewal with code from git master / 0.4.0+ before letsencrypt renew works for you. That’s because previous versions of the client would write descriptions of certificate “lineages” (series of certificates that replace each other) to /etc/letsencrypt/renew, but never update those conf files if the user switched to different settings. This has been fixed in git, and will ship in 0.4.0.

Q. What’s the new --dry-run flag?
A. The new --dry-run flag for both certonly and renew performs the certificate request(s) against the staging server, which issues test certificates that are not trusted by browsers. This verifies whether you’re apparently able to get a certificate, in your current configuration, using the method that you specified (for example, if you were using webroot authentication, whether your webroot configuration is capable of being validated by the CA). With --dry-run, the certificates obtained are not actually saved to disk and your configuration is not updated. You can use this to simulate what would apparently happen if you ran the command without --dry-run.

Q. Why does letsencrypt renew only renew some certificates?
A. letsencrypt renew has a notion of when certificates are due for renewal, currently 30 days before their expiry. We will soon document ways of changing this. Only certificates that are due for renewal get renewed, unless you specify letsencrypt renew --renew-by-default, which attempts to renewal all certificates, regardless of how old they are.

Q. Will letsencrypt renew work with the standalone plugin?
A. Yes, but you may need to stop your web server first and then start it again afterwards. If you have some cert lineages that use standalone, and some that use webroot, you won’t be able to get a single run of letsencrypt renew that works for everything at once, and may wish to keep using renewal via “certonly”

Q. Can I automate letsencrypt renew to run from cron?
A. When 0.4.0 is released, yes. Right now letsencrypt renew is being tested, so we can’t promise that nothing will change for the release.

Q. Okay, if this thing is going to be run from cron, why does it print output? cron needs it to be silent…
A. For the time being it will be best to wrap letsencrypt renew in a simple script. We considered implementing --quiet, and will probably still do so in the future, but unfortunately suppressing output in case of success isn’t a satisfactory renewal solution. The problem is that a cert renew command can have two kinds of success: (1) it succeeded because no certs were due for renewal; (2) some certs were due for renewal and it succeeded in renewing them. UNIX only supports one type of “success” return code. But in many situations those calling the renewer will need to know the difference between the two types of success because in case (2), extra legwork such as reloading TLS servers (the ones that don’t have client installer plugins) needs to happen. So for 0.4.0, we will recommend a small script to grep or redirect the output of letsencrypt renew in your cron/systemd jobs.

Q. Do renewals count against the rate limit?
A. Yes, currently renewals count against the rate limit in the same way as new certs. It is possible to hit the rate limit using letsencrypt renew and have it fail or partially fail for that reason. Rate limit policies related to renewals may be adjusted in the future.

Q. What’s this -n or --non-interactive flag?
A. --non-interactive (or -n for short) is a feature we shipped in 0.3.0. It ensures that the client will never stop to ask the user a question: if there isn’t a sensible default answer to a question, an error will be raised. This is ideal for any scripting of the client (including renewal), and also good for power users on the command line. You can use it to replace the combination of -t and --keep in previously written renewal scripts.

3 Likes
3

on Ubuntu and RedHat

cd /root/letsencrypt
git remote update
service apache2 stop
./letsencrypt-auto certonly --standalone --keep-until-expiring -d
www.example.com

... generate:

Updating letsencrypt and virtual environment dependencies...You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
..Requirement already satisfied (use --upgrade to upgrade): ConfigArgParse from git+https://github.com/kuba/ConfigArgParse.git@python2.6-0.9.3#egg=ConfigArgParse in /root/.local/share/letsencrypt/lib/python2.7/site-packages (from -r py26reqs.txt (line 2))
..
Running with virtualenv:

trying executing of

pip install -upgrade pip

ends with error no pip installed

But certificates are generated successfully.

4

Hi @petr.svec, did you also try renewing with letsencrypt renew? (That’s the part that’s totally new.)

5

Here some testing results and the hurdles I ran into. Running ./letsencrypt-auto renew resulted in some errors:

1st attempt:
...The error was Expected a numeric value for http01_port. ...
Editing the file /etc/letsencrypt/renewal/[my.domain].conf file solved this problem. I changed http01_port = None to http01_port = 80

2nd attempt:
The next error message was At least one of the (possibly) required ports is already taken..
I needed to stop the webserver: apache2ctl stop

3rd attempt
As one of the ‘early birds’ of Let’s encrypt’ I am currently stuck by the rate limit. My ISP is so kind to provide the possibility to register a subdomain and direct it to the fixed IP of the DSL. Because of the growing number of customers with a subdomain that use Let’s Encrypt I can’t renew: Error creating new cert :: Too many certificates already issued for: xs4all.nl. Skipping..
So my certificate has expired and I am not able to renew… :sob:

Before I tried the renew option I ran into some problems already posted by others, such as an inaccessible directory ./well-known:

  1. A restrictive .htaccess. Try (temporarily) removing/renaming .htaccess or better, create an alias.
  2. Insufficient rights for the web server to serve content. I ran from ./letsencrypt-auto as root while the webserver runs as user www-data and is not able to get the content in .well-known/acme-challenge. Alter permissions with chmod and chown.
6

@Randall, thanks for letting us know about your problems!

I filed

with regard to the first problem. The second problem seemed self-explanatory and in line with what we have documented (the client apparently correctly explains what you need to do in this case). For the third problem, I’ll try to talk to some of the CA folks about this issue. In principle it is covered by the existing issue

I’m sorry that your certificate expired!

7

I had a development site (ny.bryggebok.no, ny is Norwegian for new), but this site has been disabled for a while. I had a certificate for this site from the staging server, but the site is now removed from the apache and DNS configuration. Now I get the following error when running ‘letsencrypt renew’

Processing /etc/letsencrypt/renewal/foseid.no.conf
2016-02-14 21:06:28,384:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/foseid.no.conf produced an unexpected error: Failed to run Apache plugin non-interactively
Missing command line flag or config entry for this setting:
We were unable to find a vhost with a ServerName or Address of ny.bryggebok.no.
Which virtual host would you like to choose?
Choices: [‘nagios.foseid.no.conf | nagios.foseid.no | | Enabled’, ‘bryggerikart.foseid.no.conf | bryggerikart.foseid.n | | Enabled’, 'default-ssl.conf | | HTTPS | ', '000-default.conf | | | ', ‘bryggebok.no-le-ssl.conf | Multiple Names | HTTPS | Enabled’, ‘foseid.no-le-ssl.conf | Multiple Names | HTTPS | Enabled’, ‘bryggebok.no.conf | Multiple Names | | Enabled’, ‘foseid.no.conf | Multiple Names | | Enabled’, ‘nagios.foseid.no-le-ssl.conf | nagios.foseid.no | HTTPS | Enabled’, ‘bryggerikart.foseid.no-le-ssl. | bryggerikart.foseid.n | HTTPS | Enabled’, ‘web.foseid.no-le-ssl.conf | web.foseid.no | HTTPS | Enabled’, ‘web.foseid.no.conf | web.foseid.no | | Enabled’]
(The best solution is to add ServerName or ServerAlias entries to the VirtualHost directives of your apache configuration files.). Skipping.
Processing /etc/letsencrypt/renewal/web.foseid.no-0001.conf
2016-02-14 21:06:28,387:WARNING:letsencrypt.cli:An error occured while parsing /etc/letsencrypt/renewal/web.foseid.no-0001.conf. The error was Expected a numeric value for http01_port. Skipping the file.
Processing /etc/letsencrypt/renewal/web.foseid.no.conf

8

Hi, I’m using ISPConfig for a shared hosting server and had configured a cronjob (the ISPConfig plugin did that for me https://github.com/alexalouit/ISPConfig-letsencrypt )

30 02 * * * /root/.local/share/letsencrypt/bin/letsencrypt-renewer >> /var/log/ispconfig/cron.log; done

However, with that setup, the renew does not work. Manually executing the renew command results in an error:

WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/domain.de.conf produced an unexpected error: The webroot plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('Missing parts of webroot configuration; please set either --webroot-path and --domains, or --webroot-map. Run with  --help webroot for examples.',). Skipping.

I tried modifying the conf file without success - extracting the relevant parts here:

installer = apache
authenticator = webroot #(was "apache)
webroot_path = /var/www/domain.de/web/ #(was "None")
webroot = True

These changes had no effect on the result - I’m not sure what I’m missing. I still have 2 weeks left and will wait for response to this thread before manually triggering a certonly update. Of course any help with the renewal script is highly appreciated since the number of letsencypt domains is increasing every week…

Thanks!

btw: just saw that this thread is about testing the dev environment, executed again with latest git version “letsencrypt 0.5.0.dev0” - still same result

9

Seems I missed a /etc/letsencrypt/renewal/ny.bryggebok.no.conf , but this situations should probably still give a slightly better error message or provide some way to remove the renewal configuration for a domain.

10

Hi @mattanja, thanks for testing this and letting us know about your difficulties. Hopefully we can get to the bottom of this and help other people who may find themselves in a similar situation.

Could you post the exact command that you tried to use with letsencrypt renew?

As a hypothesis, do you have a line at the bottom of the renewal configuration file that says [[webroot_map]]? I’m wondering if the empty webroot_map that the client originally saved is overriding your explicit webroot setting because it’s being treated as a higher-priority setting when trying to rebuild the webroot map. :frowning: :frowning:

11

I tried using the renew option. My understanding was that letsencrypt-auto would try to remember what I had typed on the command line to get the certs and then just try to run that command again but to renew. I had to use multiple -w (for the webroot plugin) in order to get the certs I needed. letsencrypt-auto renew failed. Not only because of that, but also because it couldn’t remember what e-mail address I had given, it couldn’t remember that I accepted the TOS. I tried creating a cli.cfg in the /etc/letsencrypt directory and got passed the e-mail and TOS problem, but it didn’t work with the multiple webroot paths.

12

Apparently the renew function does not work with ECDSA certificates. If I start out with a ECDSA certificate, after renewal I get a RSA certificate, and there seems to be no config option I can change to get a ECDSA certificate.

1 Like
13

@Spork_Schivago, thanks for testing this; could you send us some more information about how you used the client before and how you ran letsencrypt-auto renew, and also exactly what error messages it printed or logged (if any)?

14

@mpqft, that's true at present -- what client and settings were you using to get your ECDSA key? Our Python letsencrypt client doesn't support ECDSA keys, so it wouldn't even have realized that you might want to use one.

1 Like
15

@mattanja, if my hypothesis about your situation is right, you should delete the [[webroot_map]] line from the bottom of your domain.de.conf file and then try the renewal again. (But can you confirm that you didn’t use webroot when you originally got the domain? You said you originally used apache? My hypothesis is based on an idea about what would happen if you tried to switch from non-webroot to webroot by editing the config file just before renewal.)

Another equivalent way of fixing it might be

letsencrypt certonly -a webroot -w /var/www/domain.de/web -d domain.de

assuming that the cert only covers domain.de and not also www.domain.de or other names. (If there are other names, they should be explicitly specified with additional -d flags at the end.) That would modify the domain.de.conf in a way that should cause plain letsencrypt renew to succeed for future renewal attempts.

16

I actually used the official letsencrypt client 0.4.0 with the webroot plugin and my own CSR. I have generated a few ECDSA certificates already. So if I understand correctly, it’s currently not possible to do letsencrypt renew with custom CSR?

17

@mpqft, oh, I didn’t think about the custom CSR method at all. No, if you do letsencrypt renew with a custom CSR you will get

        raise errors.Error("Currently, the renew verb cannot be used when "
                           "specifying a CSR file. Please try the certonly "
                           "command instead.")

As the error message will suggest, you might want to try certonly; I think the differences between the two are unlikely to affect your use case. There is no difference on the CA back end between certonly and renew; the CA doesn’t record your request as a “renewal” in one case but not the other, it just has to do with how the client determines its own configuration when requesting the certificate.

You can add --renew-by-default (which we are renaming to --force-renew) if you want to skip certonly's interactive prompt about whether to save the new cert as a replacement for an existing cert.

Edit: one limitation of certonly --force-renew is that it’s not able to determine when the existing certificate is near expiry, so if you ran this automatically every day, you would quickly hit the rate limit because a new certificate would be requested every day. We don’t have any functionality implemented currently that combines checking when an existing certificate is near expiry with using a specified CSR file for the new certificate request. Hopefully we’ll either add this soon or some other cert management features that we implement will be more useful for scripting this check yourself.

1 Like
18

@schoen Thank you! Yes, I think you're right. After removing the [[webroot_map]] line at the bottom of the renewal config I got a different error (not finding the correct credentials in the webroot).

2016-02-20 00:36:05,190:WARNING:letsencrypt.cli:Attempting to renew cert from /etc/letsencrypt/renewal/domain.com.conf produced an unexpected error: Failed authorization procedure. domain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://domain.com/.well-known/acme-challenge/xxxx-xxxx [x.x.x.x]: 404. Skipping.

However, executing the certonly for the same domain worked fine
/opt/letsencrypt/letsencrypt-auto certonly --keep-until-expiring -w /var/www/domain.com/web -d domain.com

I may still have some misconfiguration in my renewal files and will continue testing as soon as other certs are due.

For the renewal files being generated for new certificates I think the content was not generated correctly according to the way the original cert was created. In any case it would help a lot to have a documentation for the parameters that can be used there - I couldn't seem to find a page discribing the format. Thanks!

Edit: I only now realized that the renewal file will be updated by such an action - which is good to know. The part that I probably got wrong (missing documentation?) is that the webroot_path must end with a comma (?) and/or that the entry was added to the [[webroot_map]] part:

webroot_path = /var/www/domain.com/web, [[webroot_map]] domain.com = /var/www/domain.com/web

19

Ugh… testing that was a mistake.

I only have 5 certs. 3 were issued new earlier today. Then I went and ran “renew” because my two old certs are about to expire. Only I was Instantly rate limited without the oldest certs actually renewing. That cert will now likely expire before the rate-limit resets :frowning:

Renew should be intelligent and operate on certs from oldest to newest… I can’t even tell what order it goes in, it doesn’t appear alphabetical or date.

Really though that shouldn’t matter, the rate limits are just too low for renewals. It wouldn’t be so bad if we had wildcard certs, but with every sub-domain being a separate cert, five is not sufficient specifically because of the renewal issue.

Soon I’ll have 6 sub-domains and frankly I have better things to do than login to my server twice a week apart to renew them every 60 days.

1 Like
20

You know you can request SAN certs, right? as in, multiple domains on one cert, counting as just one against the rate limit? Just specify multiple -d options on the command line.

1 Like