Renewal fails on cubietruck (due to no space left on device?)


#1

Hi there, I can’t renew my letsencrypt-certificate which will expire in two days. Up to now, everything is working fine and I can reach my nextcloud-server via https without problems. When I try to renew the certificate I get a very long output (see below) that I don’t understand enough to find the mistake. Is it possible, that there is not enough space left on the SD-card in the cubietruck?

I’d be thankful for advice.

My domain is: meinserver.de

I ran this command: letsencrypt certonly -a webroot --webroot-path=/var/www/letsencrypt --rsa-key-size 4096 -d meinserver.de

It produced this output: see below

My web server is (include version): cubietruck, nginx (newest version 1.10.3-0ubuntu0.16.04.2)

The operating system my web server runs on is (include version): ARMBIAN 5.38 stable Ubuntu 16.04.4 LTS 3.4.113-sun7i

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes, headless via ssh

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no


some of the output produced by the above command:

Message: ‘Received response:\nHTTP %d\n%s\n\n%s’
Arguments: (200, ‘Server: nginx\nContent-Type: application/json\nContent-Length: 1737\nLink: https://acme-v01.api.letsencrypt.org/acme/new-cert;rel=“next”\nReplay-Nonce: D3a0nqmp7SRUwSnm5k5_3LCjEuBYAYNFYVXKKW1LZt0\nX-Frame-Options: DENY\nStrict-Transport-Security: max-age=604800\nExpires: Tue, 03 Apr 2018 16:19:47 GMT\nCache-Control: max-age=0, no-cache, no-store\nPragma: no-cache\nDate: Tue, 03 Apr 2018 16:19:47 GMT\nConnection: keep-alive’, b’{\n “identifier”: {\n “type”: “dns”,\n “value”: “meinserver.de”\n },\n “status”: “invalid”,\n “expires”: “2018-04-10T16:19:43Z”,\n “challenges”: [\n {\n “type”: “dns-01”,\n “status”: “invalid”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/aFzaBc89km-Uy_F9XG_zBt7ohlAHHNaoWXK0rgOguOU/4074722061”,\n “token”: “QKVnAztBMnrcueFcUWmRTk_Id5GN8PhsrzQDWuluDrs”\n },\n {\n “type”: “http-01”,\n “status”: “invalid”,\n “error”: {\n “type”: “urn:acme:error:connection”,\n “detail”: “Fetching http://meinserver.de/.well-known/acme-challenge/yVJ9oAOly25VNF86-ML4cfp3Wj0yb1N38J_PjwhsCXY: Error getting validation data”,\n “status”: 400\n },\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/aFzaBc89km-Uy_F9XG_zBt7ohlAHHNaoWXK0rgOguOB/4074722062”,\n “token”: “yVJ9oAOly25VNF86-ML4cfp3Wj0yb1R38J_PjwhsCXY”,\n “keyAuthorization”: “yVJ9oAOly25VNF86-ML4cfp3Wj0yb1N38J_PjwhsCXY.nCuWy5cc7Fh1Lh03AghrCbFvMfZtIpScRYe4p38U3GQ”,\n “validationRecord”: [\n {\n “url”: “http://meinserver.de/.well-known/acme-challenge/yVJ9oAOly25VNF86-ML4cfp3Wj0yb1R38J_PjwhsCXY”,\n “hostname”: “meinserver.de”,\n “port”: “80”,\n “addressesResolved”: [\n “92.196.51.134”\n ],\n “addressUsed”: “92.196.51.134”\n }\n ]\n },\n {\n “type”: “tls-sni-01”,\n “status”: “invalid”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/aFzaBc89km-Uy_F9XG_zBt7ohlAHHNaoWXK0rgOguOU/4074722063”,\n “token”: “TAiVigglcavRdvdszSyVNzUlSVWkH2bm20L7o5TgLyo”\n }\n ],\n “combinations”: [\n [\n 0\n ],\n [\n 1\n ],\n [\n 2\n ]\n ]\n}’)
— Logging error —
Traceback (most recent call last):
File “/usr/lib/python3.5/logging/handlers.py”, line 71, in emit
if self.shouldRollover(record):
File “/usr/lib/python3.5/logging/handlers.py”, line 188, in shouldRollover
self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
OSError: [Errno 28] No space left on device

[…]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.5/logging/handlers.py”, line 71, in emit
if self.shouldRollover(record):
File “/usr/lib/python3.5/logging/handlers.py”, line 188, in shouldRollover
self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
OSError: [Errno 28] No space left on device
Call stack:
File “/usr/bin/letsencrypt”, line 11, in
load_entry_point(‘certbot==0.22.2’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1266, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1157, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 113, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/usr/lib/python3/dist-packages/certbot/renewal.py”, line 297, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 294, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 330, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 79, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 156, in _respond
self._cleanup_challenges(aauthzrs, active_achalls)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 303, in _cleanup_challenges
self.auth.cleanup(achalls)
File “/usr/lib/python3/dist-packages/certbot/plugins/webroot.py”, line 232, in cleanup
logger.debug(“All challenges cleaned up”)
Message: ‘All challenges cleaned up’
Arguments: ()
— Logging error —
Traceback (most recent call last):
File “/usr/lib/python3.5/logging/handlers.py”, line 71, in emit
if self.shouldRollover(record):
File “/usr/lib/python3.5/logging/handlers.py”, line 188, in shouldRollover
self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
OSError: [Errno 28] No space left on device
Call stack:
File “/usr/lib/python3/dist-packages/certbot/log.py”, line 327, in post_arg_parse_except_hook
logger.debug(‘Exiting abnormally:’, exc_info=exc_info)
Message: ‘Exiting abnormally:’
Arguments: ()
Failed authorization procedure. meinserver.de (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://meinserver.de/.well-known/acme-challenge/yVJ9oAOly25VNF86-ML4cfp3Wj0yb1N38J_PjwhsCXY: Error getting validation data

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: meinserver.de
    Type: connection
    Detail: Fetching
    http://meinserver.de/.well-known/acme-challenge/yVJ9oAOly25VNF86-ML4cfp3Wj0yb1N38J_PjwhsCXY:
    Error getting validation data

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.



#2

Could you investigate this by running df and df -i?


#3

df

Filesystem 1K-blocks Used Available Use% Mounted on
udev 955168 0 955168 0% /dev
tmpfs 201388 3252 198136 2% /run
/dev/mmcblk0p1 15164272 2919412 12062960 20% /
tmpfs 1006932 0 1006932 0% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 1006932 0 1006932 0% /sys/fs/cgroup
tmpfs 1006932 1756 1005176 1% /usr/local/tmp/sessions
tmpfs 1006932 8 1006924 1% /usr/local/tmp/cache
tmpfs 1006932 16 1006916 1% /tmp
tmpfs 1006932 0 1006932 0% /var/tmp
/dev/sda1 480720596 188221544 268079776 42% /var/nc_data
log2ram 51200 51200 0 100% /var/log
tmpfs 201388 0 201388 0% /run/user/114
tmpfs 201388 0 201388 0% /run/user/1000

df -i

Filesystem Inodes IUsed IFree IUse% Mounted on
udev 147857 442 147415 1% /dev
tmpfs 173739 580 173159 1% /run
/dev/mmcblk0p1 932672 140078 792594 16% /
tmpfs 173739 1 173738 1% /dev/shm
tmpfs 173739 4 173735 1% /run/lock
tmpfs 173739 11 173728 1% /sys/fs/cgroup
tmpfs 173739 440 173299 1% /usr/local/tmp/sessions
tmpfs 173739 15 173724 1% /usr/local/tmp/cache
tmpfs 173739 22 173717 1% /tmp
tmpfs 173739 5 173734 1% /var/tmp
/dev/sda1 30531584 143876 30387708 1% /var/nc_data
log2ram 173739 375 173364 1% /var/log
tmpfs 173739 5 173734 1% /run/user/114
tmpfs 173739 5 173734 1% /run/user/1000


#4

/var/log is 50 MiB, and entirely full. You need to work that out with your OS and log2ram. I’m not familiar with it, but apparently it can be made larger in the configuration file. Or you can delete stuff, adjust log rotation, adjust logging, etc.

Certbot writes to /var/log/letsencrypt/ by default, so it’s no surprise it ran into issues.

Still, that may or may not be why validation failed.


#5

Thanks a lot for the fast response. I deleted old log-files and retried. The error “no space left on device” is gone now and the new error-message is a lot shorter. But unfortunately the certificate still does not renew. The full error massage now is:

: letsencrypt certonly -a webroot --webroot-path=/var/www/letsencrypt --rsa-key-size 4096 -d meinserver.de


Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for meinserver.de
Using the webroot path /var/www/letsencrypt for all unmatched domains.
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. meinserver.de (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://meinserver.de/.well-known/acme-challenge/0J8swv6NDCwnsYc9gBORAVKSxRXc5WyLfxoNKrdzTDs: Error getting validation data

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: meinserver.de
    Type: connection
    Detail: Fetching
    http://meinserver.de/.well-known/acme-challenge/0J8swv6NDCwnsYc9gBORAVKSxRXc5WyLfxoNKrdzTDs:
    Error getting validation data

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.



#6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.