Please help to determine what happened to certbot

My domain is: https://tvoydom.info, https://vostok-zapad.kharkov.ua, https://en.aviatranslation.com

I ran this command: certbot auto

It produced this output:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.31.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1333, in main
log.pre_arg_parse_setup()
File “/usr/lib/python3/dist-packages/certbot/log.py”, line 55, in pre_arg_parse_setup
temp_handler = TempHandler()
File “/usr/lib/python3/dist-packages/certbot/log.py”, line 243, in init
stream = tempfile.NamedTemporaryFile(‘w’, delete=False)
File “/usr/lib/python3.5/tempfile.py”, line 679, in NamedTemporaryFile
prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
File “/usr/lib/python3.5/tempfile.py”, line 269, in _sanitize_params
dir = gettempdir()
File “/usr/lib/python3.5/tempfile.py”, line 435, in gettempdir
tempdir = _get_default_tempdir()
File “/usr/lib/python3.5/tempfile.py”, line 370, in _get_default_tempdir
dirlist)
FileNotFoundError: [Errno 2] No usable temporary directory found in [’/tmp’, ‘/var/tmp’, ‘/usr/tmp’, ‘/root’]

My web server is (include version): Apache/2.4.38 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 16.04.6 LTS

My hosting provider, if applicable, is: Scaleway, VPS.

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): Pytty/WinSCP

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): I cannot determine it - certbot does not work properly since this morning.

=================================================

All was OK till this morning when I discovered that all my sites are not available. I tried to uderstand what is a problem, launched WinSCP/Putty and then found that something happened to Certbot - see my data above.

Need an advice - what should I do?

Hi @trommo

programs need space to create temporary files. Looks like your harddisk is corrupt or there is no free space.

What says

ls -al /tmp

Same with the other directories.

root@trommo:/# ls -al /tmp
total 36
drwxrwxrwt 9 root root 4096 Jun 7 04:00 .
drwxr-xr-x 22 root root 4096 Mar 1 23:06 …
drwxrwxrwt 2 root root 4096 May 3 12:22 .font-unix
drwxrwxrwt 2 root root 4096 May 3 12:22 .ICE-unix
drwx------ 3 root root 4096 May 26 21:01 systemd-private-69411ad0b430402da6b4edf1f5bee32e-apache2.service-bRzmYF
drwx------ 3 root root 4096 May 3 12:22 systemd-private-69411ad0b430402da6b4edf1f5bee32e-openvpn@server.service-5I4X4c
drwxrwxrwt 2 root root 4096 May 3 12:22 .Test-unix
drwxrwxrwt 2 root root 4096 May 3 12:22 .X11-unix
drwxrwxrwt 2 root root 4096 May 3 12:22 .XIM-unix
root@trommo:/#

I did not find folder tmp in /usr… Is this important?

I don’t know. Is there enough free space? /tmp exists, that’s good. Create the missing directories, but if there isn’t enough free space, that will not help.

When I tried to create folder tmp in /usr in WinSCP, I’ve got an error:

General failure (server should provide error description).
Error code: 4
Error message from server: Failure

Common reasons for the Error code 4 are:

  • Renaming a file to a name of already existing file.
  • Creating a directory that already exists.
  • Moving a remote file to a different filesystem (HDD).
  • Uploading a file to a full filesystem (HDD).
  • Exceeding a user disk quota.

But when I run df -h in console, I’ve got the following message:

root@trommo:/# df -h
Filesystem Size Used Avail Use% Mounted on
none 2.0G 0 2.0G 0% /dev
tmpfs 396M 54M 342M 14% /run
/dev/vda 46G 7.0G 37G 17% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 396M 0 396M 0% /run/user/0

As I understood - I have 37 Gb of free space on disk? Am I right?

What about “df -hi”?

root@trommo:/# df -hi
Filesystem Inodes IUsed IFree IUse% Mounted on
none 494K 1.3K 492K 1% /dev
tmpfs 494K 1.5K 493K 1% /run
/dev/vda 3.0M 128K 2.8M 5% /
tmpfs 494K 1 494K 1% /dev/shm
tmpfs 494K 4 494K 1% /run/lock
tmpfs 494K 15 494K 1% /sys/fs/cgroup
cgmfs 494K 12 494K 1% /run/cgmanager/fs
tmpfs 494K 4 494K 1% /run/user/0
root@trommo:/#

Did you run certbot as root or sudo?

Is this an error?

certbot

or

certbot-auto

The same message as I described in start post.

what happens if you do a command that should never fail (if everything is normal at the hardware level)

echo “aaaaa” >/tmp/lerulez

for example ?

What says

printenv

Is a temp directory defined?

The response is:

root@trommo:~# echo “aaaaa” >/tmp/lerulez
-bash: /tmp/lerulez: Read-only file system

But the /tmp folder has the following rights: rwxrwxrwt, as usual…

1 Like

The response is next:

root@trommo:~# printenv
XDG_SESSION_ID=c141
TERM=xterm
SHELL=/bin/bash
SSH_CLIENT=85.90.216.199 50793 22
SSH_TTY=/dev/pts/0
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd =40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;4 4:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01; 31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7 z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=0 1;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz =01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31: .rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;3 1:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm =01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35: .png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=0 1;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:* .m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01; 35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl= 01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.o gv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36 :.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=0 0;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/us r/local/games
PWD=/root
LANG=en_US.UTF-8
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=85.90.216.199 50793 10.2.211.9 22
LESSOPEN=| /usr/bin/lesspipe %s
XDG_RUNTIME_DIR=/run/user/0
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/printenv
root@trommo:~#

1 Like

readonly write system while writing to /tmp suggest strongly there is something very wrong with your VPS. Probably hardware problem. Escalate the issue with your provider.

1 Like

Yep, looks like a general problem of your system. So Certbot can’t create new files.

Yes, the problem was in the SSD on server…

2 Likes

Sorry to hear that. :frowning_face:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.