Letsencrypt error: cannot change profile for the next exec call: No such file or directory

robbrandt · November 2, 2021, 4:27pm

.....
port 80 namevhost awards.botany.org (/etc/apache2/sites-enabled/awards.botany.org.conf:1)
alias www.awards.botany.org
alias webmail.awards.botany.org
alias admin.awards.botany.org
.....
port 443 namevhost awards.botany.org (/etc/apache2/sites-enabled/awards.botany.org.conf:65)
alias www.awards.botany.org
alias webmail.awards.botany.org
alias admin.awards.botany.org

rg305 · November 2, 2021, 4:29pm

So, will it do this now?:
certbot certificates

and can we see this file?:

robbrandt · November 2, 2021, 4:46pm

$ sudo certbot certificates
cannot change profile for the next exec call: No such file or directory

<VirtualHost *:80>
SuexecUserGroup "#1007" "#1006"
ServerName awards.botany.org
ServerAlias www.awards.botany.org
ServerAlias webmail.awards.botany.org
ServerAlias admin.awards.botany.org
DocumentRoot /home/botany/domains/awards.botany.org/public_html
ErrorLog /var/log/virtualmin/awards.botany.org_error_log
CustomLog /var/log/virtualmin/awards.botany.org_access_log "combined"
ScriptAlias /cgi-bin/ /home/botany/domains/awards.botany.org/cgi-bin/
ScriptAlias /awstats/ /home/botany/domains/awards.botany.org/cgi-bin/
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/botany/domains/awards.botany.org/public_html>
Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted
AddType application/x-httpd-php .php
AddHandler fcgid-script .php
AddHandler fcgid-script .php5.6
AddHandler fcgid-script .php7.0
AddHandler fcgid-script .php7.1
AddHandler fcgid-script .php7.2
AddHandler fcgid-script .php7.3
AddHandler fcgid-script .php7.4
AddHandler fcgid-script .php8.0
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.2.fcgi .php
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php5.6.fcgi .php5.6
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.0.fcgi .php7.0
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.1.fcgi .php7.1
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.2.fcgi .php7.2
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.3.fcgi .php7.3
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.4.fcgi .php7.4
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php8.0.fcgi .php8.0
</Directory>
<Directory /home/botany/domains/awards.botany.org/cgi-bin>
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.awards.botany.org
RewriteRule ^(.*) https://awards.botany.org:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.awards.botany.org
RewriteRule ^(.*) https://awards.botany.org:10000/ [R]
<Files awstats.pl>
AuthName "awards.botany.org statistics"
AuthType Basic
AuthUserFile /home/botany/domains/awards.botany.org/.awstats-htpasswd
require valid-user
</Files>
LogLevel warn
RemoveHandler .php
RemoveHandler .php5.6
RemoveHandler .php7.0
RemoveHandler .php7.1
RemoveHandler .php7.2
RemoveHandler .php7.3
RemoveHandler .php7.4
RemoveHandler .php8.0
php_admin_value engine Off
IPCCommTimeout 9999
FcgidMaxRequestLen 1073741824
</VirtualHost>
<VirtualHost *:443>
SuexecUserGroup "#1007" "#1006"
ServerName awards.botany.org
ServerAlias www.awards.botany.org
ServerAlias webmail.awards.botany.org
ServerAlias admin.awards.botany.org
DocumentRoot /home/botany/domains/awards.botany.org/public_html
ErrorLog /var/log/virtualmin/awards.botany.org_error_log
CustomLog /var/log/virtualmin/awards.botany.org_access_log "combined"
ScriptAlias /cgi-bin/ /home/botany/domains/awards.botany.org/cgi-bin/
ScriptAlias /awstats/ /home/botany/domains/awards.botany.org/cgi-bin/
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/botany/domains/awards.botany.org/public_html>
Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted
AddType application/x-httpd-php .php
AddHandler fcgid-script .php
AddHandler fcgid-script .php5.6
AddHandler fcgid-script .php7.0
AddHandler fcgid-script .php7.1
AddHandler fcgid-script .php7.2
AddHandler fcgid-script .php7.3
AddHandler fcgid-script .php7.4
AddHandler fcgid-script .php8.0
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.2.fcgi .php
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php5.6.fcgi .php5.6
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.0.fcgi .php7.0
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.1.fcgi .php7.1
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.2.fcgi .php7.2
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.3.fcgi .php7.3
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php7.4.fcgi .php7.4
FCGIWrapper /home/botany/domains/awards.botany.org/fcgi-bin/php8.0.fcgi .php8.0
</Directory>
<Directory /home/botany/domains/awards.botany.org/cgi-bin>
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
Require all granted
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.awards.botany.org
RewriteRule ^(.*) https://awards.botany.org:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.awards.botany.org
RewriteRule ^(.*) https://awards.botany.org:10000/ [R]
<Files awstats.pl>
AuthName "awards.botany.org statistics"
AuthType Basic
AuthUserFile /home/botany/domains/awards.botany.org/.awstats-htpasswd
require valid-user
</Files>
LogLevel warn
SSLEngine on
SSLCertificateFile /home/botany/domains/awards.botany.org/ssl.cert
SSLCertificateKeyFile /home/botany/domains/awards.botany.org/ssl.key
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCACertificateFile /home/botany/domains/awards.botany.org/ssl.ca
RemoveHandler .php
RemoveHandler .php5.6
RemoveHandler .php7.0
RemoveHandler .php7.1
RemoveHandler .php7.2
RemoveHandler .php7.3
RemoveHandler .php7.4
RemoveHandler .php8.0
php_admin_value engine Off
IPCCommTimeout 9999
FcgidMaxRequestLen 1073741824
</VirtualHost>

MikeMcQ · November 2, 2021, 4:47pm

@robbrandt

Since certbot still fails to run please show results of this:
sudo aa-status

Then try this TEST:

sudo systemctl stop apparmor
certbot certificates

If certbot now runs we have identified apparmor as the problem and can take further steps to correct it.

Update: I thought I just saw rob post that certbot did not run again. But, I guess not since the post is missing. So, only try above if certbot still does not run.

rg305 · November 2, 2021, 4:56pm

Is AA the latest version?

MikeMcQ · November 2, 2021, 4:58pm

I believe so.
https://help.ubuntu.com/community/AppArmor#Usage

rg305 · November 2, 2021, 5:01pm

Maybe my wires are crossed...
But I don't see a version number in this topic, nor having done "apt update && apt upgrade".

robbrandt · November 2, 2021, 5:13pm

$ sudo systemctl stop apparmor
$ sudo certbot certificates
cannot change profile for the next exec call: No such file or directory

Akismet held up one of my posts so that might explain you seeing it then not seeing it. It's posted now.

robbrandt · November 2, 2021, 5:16pm

I don't see a version switch on the Usage page, and aa-status doesn't report the version. But it appears to be:

MikeMcQ · November 2, 2021, 5:19pm

What did aa-status report?

rg305 · November 2, 2021, 5:19pm

That does seem to be the latest version.
hmm...

robbrandt · November 2, 2021, 5:20pm

apt update && apt upgrade

Hit:1 Index of /ubuntu focal InRelease
Hit:2 Index of /ubuntu focal-updates InRelease
Hit:3 Index of /ubuntu focal-backports InRelease
Get:4 Index of /ubuntu focal-security InRelease [114 kB]
Hit:5 Index of /232905/apt/ubuntu bionic InRelease
Hit:6 Index of /ondrej/php/ubuntu focal InRelease
Ign:7 Index of /download/repository sarge InRelease
Hit:8 Index of /vm/6/gpl/apt virtualmin-universal InRelease
Hit:9 Index of /vm/6/gpl/apt virtualmin-focal InRelease
Err:10 Index of /download/repository sarge Release
Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 104.207.151.13 443]
Reading package lists... Done
E: The repository 'Index of /download/repository sarge Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

So I can see I have a webmin update issue but I doubt that's related to the current problem, and I have the most recent versions of webmin/virtualmin.

MikeMcQ · November 2, 2021, 5:26pm

Well, that site returns an odd cert chain but I agree I do not think that is causing this problem.

Certificate chain
 0 s:/CN=download.webmin.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
 3 s:/C=US/O=Let's Encrypt/CN=R3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

That site operator needs to correct that

rg305 · November 2, 2021, 5:29pm

Agreed 0,1,2 are good, and should be all that is served.
#3 expired on 2021/09/21 and has no business being served.

The name has two IPs:

Name:      download.webmin.com
Addresses: 104.207.151.13
           108.60.199.109

The other one looks correct:

echo | openssl s_client -connect 108.60.199.109:443 | head
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = jamie.cloud.virtualmin.com
verify return:1
CONNECTED(00000005)
---
Certificate chain
 0 s:CN = jamie.cloud.virtualmin.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

robbrandt · November 2, 2021, 5:30pm

$ sudo aa-status
apparmor module is loaded.
22 profiles are loaded.
22 profiles are in enforce mode.
/usr/bin/freshclam
/usr/bin/lxc-start
/usr/bin/man
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/NetworkManager/nm-dhcp-helper
/usr/lib/connman/scripts/dhclient-script
/usr/lib/snapd/snap-confine
/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/sbin/clamd
/usr/sbin/mysqld
/usr/sbin/named
/usr/sbin/tcpdump
/{,usr/}sbin/dhclient
lsb_release
lxc-container-default
lxc-container-default-cgns
lxc-container-default-with-mounting
lxc-container-default-with-nesting
man_filter
man_groff
nvidia_modprobe
nvidia_modprobe//kmod
0 profiles are in complain mode.
4 processes have profiles defined.
4 processes are in enforce mode.
/usr/bin/freshclam (902)
/usr/sbin/mysqld (1082)
/usr/sbin/named (170608)
/sbin/dhclient (683) /{,usr/}sbin/dhclient
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

robbrandt · November 2, 2021, 5:33pm

I have update/upgrade running on a cron regularly, I get notifications when it fails. So everything should be up to date pretty much always.

rg305 · November 2, 2021, 5:42pm

Does your cron tell you that happens?
I'm not saying that fixing that will fix your problem.
I'm saying you can't be certain that webmin hasn't posted some update - that might just fix this problem.

If you want to temporarily bypass that problem, add this line to your /etc/hosts file:
108.60.199.109 download.webmin.com
Then retry the update/upgrade.
Once completed, you can remove the line.

MikeMcQ · November 2, 2021, 6:03pm

Well, it looks like AA is protecting snapd. AA is known to produce the exact error message you see and we now see it has a profile for snapd. So, still a strong candidate as primary cause.

Did you start AA again after stopping it for the test earlier? This aa-status command looks like it is still active. I would have thought stopping it would have kept it stopped until a reboot (since we did not also disable it in systemd)

In any event, we can try to disable the profiles for snapd. I am just not certain of the names to use following the instructions here. That is, should it include the path shown in aa-status or not? Is there a forum for AppArmor? I think they could better help you. At least for me I am just reading their docs
https://help.ubuntu.com/community/AppArmor#Usage
and walking thru the steps

Some of the paths in the aa-status also look funny - see the dhclient path and also the double slash for snapd. Perhaps looking at AA log would shed light on problems too:
sudo aa-logprof

robbrandt · November 2, 2021, 6:07pm

Only if there's an install error.

"cannot change profile for the next exec call: No such file or directory" error happens from command line, so webmin not involved. Webmin isn't like Cpanel that alters the distro. The webmin people are aware of my problem and say no one else is reporting it, and haven't seen it before. My webmin and virtualmin are up to date.

I must say I am surprised that there doesn't seem to be a way to tell what's missing in "No such file or directory". What file? What directory? There must be a log somewhere? And which profile can't be changed?

MikeMcQ · November 2, 2021, 6:09pm

Yes, see my post 38 which I submitted while you were in process