Yeah, that was what I was expecting. The /var/log/letsencrypt/letsencrypt.log contents are the ones that are interesting to show where the execution halted at.
Yes, I see.
At the moment, /var/lib/letsencrypt has only the backup directory in it, so the apache configtest is going to fail.
ls -la /var/lib/letsencrypt/
total 12
drwxr-xr-x 3 root root 4096 Nov 20 13:21 .
drwxr-xr-x. 32 root root 4096 Nov 18 14:12 …
drwxr-xr-x 2 root root 4096 Nov 18 14:12 backups
Ok, so there is more of the leftover broken state in place. You should be able to find the references by running grep -Rn "/var/lib/letsencrypt" /etc/httpd/. These were added to the Apache configuration by Certbot most likely before the abnormal shutdown. I still have no idea what broke your temp_checkpoint however, but I guess we get there when we are able to restore a clean state.
So next thing is to remove the references to /var/lib/letsencrypt from whatever file under httpd configuration it exists.
All the references are in the /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf file (and the httpd error log):
sudo grep -Rn “/var/lib/letsencrypt” /etc/httpd/
/etc/httpd/logs/error_log:23:[Sun Nov 19 07:44:43.458125 2017] [authz_core:error] [pid 2451] [client 157.55.39.192:6210] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:24:[Sun Nov 19 07:44:43.534394 2017] [authz_core:error] [pid 2451] [client 157.55.39.192:6210] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:25:[Sun Nov 19 07:44:43.610569 2017] [authz_core:error] [pid 2451] [client 157.55.39.192:6210] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:432:[Sun Nov 19 09:15:15.861091 2017] [authz_core:error] [pid 1117] [client 163.172.4.153:24906] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page, referer: https://rlewischocolatier.com/
/etc/httpd/logs/error_log:435:[Sun Nov 19 09:32:01.877556 2017] [authz_core:error] [pid 1263] [client 180.76.15.32:33024] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:437:[Sun Nov 19 09:37:50.562495 2017] [authz_core:error] [pid 6558] [client 66.249.64.13:46453] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:438:[Sun Nov 19 09:37:50.970077 2017] [authz_core:error] [pid 1251] [client 66.249.64.9:50533] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:439:[Sun Nov 19 09:58:08.783375 2017] [authz_core:error] [pid 1266] [client 66.249.64.137:36965] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:440:[Sun Nov 19 09:58:09.109342 2017] [authz_core:error] [pid 7958] [client 66.249.64.139:37024] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:441:[Sun Nov 19 10:03:10.141109 2017] [authz_core:error] [pid 1118] [client 66.249.64.139:43061] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:523:[Sun Nov 19 10:48:10.739607 2017] [authz_core:error] [pid 1266] [client 66.249.64.9:63217] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:524:[Sun Nov 19 10:48:41.107970 2017] [authz_core:error] [pid 7958] [client 66.249.64.11:50429] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page, referer: https://www.rlewischocolatier.com/
/etc/httpd/logs/error_log:606:[Sun Nov 19 11:27:11.943679 2017] [authz_core:error] [pid 1261] [client 178.154.200.29:40386] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:607:[Sun Nov 19 11:27:15.488294 2017] [authz_core:error] [pid 1261] [client 178.154.200.29:40386] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:4474:[Sun Nov 19 14:47:46.375740 2017] [authz_core:error] [pid 27165] [client 174.138.61.43:53582] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page, referer: http://45.33.78.85:80/
/etc/httpd/logs/error_log:4475:[Sun Nov 19 14:51:32.341010 2017] [authz_core:error] [pid 1287] [client 180.76.15.6:36228] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:5470:[Sun Nov 19 17:13:53.691971 2017] [authz_core:error] [pid 1260] [client 180.76.15.140:28519] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:5552:[Sun Nov 19 17:15:22.257570 2017] [authz_core:error] [pid 31164] [client 180.76.15.144:53332] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:6042:[Sun Nov 19 17:39:21.545777 2017] [authz_core:error] [pid 5305] [client 180.76.15.159:47284] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:7354:[Sun Nov 19 18:57:26.503258 2017] [authz_core:error] [pid 7957] [client 180.76.15.17:46391] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:7356:[Sun Nov 19 19:00:00.416228 2017] [authz_core:error] [pid 27146] [client 178.154.200.29:52426] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:7357:[Sun Nov 19 19:00:03.959475 2017] [authz_core:error] [pid 27146] [client 178.154.200.29:52426] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:8090:[Sun Nov 19 20:25:42.240675 2017] [authz_core:error] [pid 31163] [client 180.76.15.160:25886] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:9312:[Sun Nov 19 21:05:29.078182 2017] [authz_core:error] [pid 9681] [client 180.76.15.14:16885] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:9319:[Sun Nov 19 21:38:48.908939 2017] [authz_core:error] [pid 1266] [client 180.76.15.154:36507] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:9320:[Sun Nov 19 21:39:03.133918 2017] [authz_core:error] [pid 1263] [client 180.76.15.137:52875] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11277:[Mon Nov 20 00:52:30.436642 2017] [authz_core:error] [pid 9677] [client 180.76.15.21:54639] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11362:[Mon Nov 20 01:36:47.392975 2017] [authz_core:error] [pid 7951] [client 180.76.15.136:50075] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11363:[Mon Nov 20 01:44:32.181599 2017] [authz_core:error] [pid 5306] [client 171.25.193.78:64392] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11364:[Mon Nov 20 01:44:35.456654 2017] [authz_core:error] [pid 1266] [client 171.25.193.78:10912] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11365:[Mon Nov 20 01:44:40.730065 2017] [authz_core:error] [pid 3725] [client 51.15.196.158:43250] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11366:[Mon Nov 20 01:44:46.947880 2017] [authz_core:error] [pid 9681] [client 51.15.196.158:45832] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11367:[Mon Nov 20 01:44:50.752411 2017] [authz_core:error] [pid 9677] [client 163.172.151.47:46144] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11368:[Mon Nov 20 01:44:54.287443 2017] [authz_core:error] [pid 31871] [client 163.172.151.47:46276] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11369:[Mon Nov 20 01:44:59.108917 2017] [authz_core:error] [pid 1262] [client 37.187.129.166:50834] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11451:[Mon Nov 20 02:32:30.558202 2017] [authz_core:error] [pid 31871] [client 180.76.15.140:49500] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11621:[Mon Nov 20 04:12:31.031869 2017] [authz_core:error] [pid 1291] [client 180.76.15.18:29693] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11637:[Mon Nov 20 06:05:25.913145 2017] [authz_core:error] [pid 31163] [client 180.76.15.140:11124] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11662:[Mon Nov 20 08:57:11.142866 2017] [authz_core:error] [pid 31879] [client 66.249.64.135:41461] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:11663:[Mon Nov 20 08:57:11.165551 2017] [authz_core:error] [pid 31879] [client 66.249.64.135:41461] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:12913:[Mon Nov 20 12:12:29.002316 2017] [authz_core:error] [pid 1116] [client 180.76.15.15:44302] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:12914:[Mon Nov 20 12:24:53.698435 2017] [authz_core:error] [pid 1266] [client 178.154.200.29:49784] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:12915:[Mon Nov 20 12:24:57.175709 2017] [authz_core:error] [pid 1266] [client 178.154.200.29:49784] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:12921:[Mon Nov 20 12:43:00.093574 2017] [authz_core:error] [pid 31878] [client 180.76.15.140:23369] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:13006:[Mon Nov 20 13:13:47.797025 2017] [authz_core:error] [pid 1258] [client 157.55.39.209:17018] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:13011:[Mon Nov 20 13:57:20.024728 2017] [authz_core:error] [pid 31881] [client 87.250.224.81:41756] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:13012:[Mon Nov 20 13:57:20.248267 2017] [authz_core:error] [pid 9680] [client 178.154.200.29:37054] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:13013:[Mon Nov 20 13:57:23.689126 2017] [authz_core:error] [pid 9680] [client 178.154.200.29:37054] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log:13015:[Mon Nov 20 13:57:25.196502 2017] [authz_core:error] [pid 27146] [client 178.154.200.29:37744] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log-20171119:26146:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:26147:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:26148:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:26149:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:26150:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:26151:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:26193:[Wed Nov 15 10:02:19.035782 2017] [authz_core:error] [pid 2497] [client 180.76.15.156:41682] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log-20171119:26198:[Wed Nov 15 10:08:56.060205 2017] [authz_core:error] [pid 2471] [client 180.76.15.18:36674] AH01630: client denied by server configuration: /var/lib/letsencrypt/tls_sni_01_page
/etc/httpd/logs/error_log-20171119:57606:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:57607:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:57608:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:57609:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:57610:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/logs/error_log-20171119:57611:AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:10: SSLCertificateFile /var/lib/letsencrypt/spv3X8QfNaFl_73xCG55FPVCmykQnfPsSPBNIWHdoho.crt
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:11: SSLCertificateKeyFile /var/lib/letsencrypt/spv3X8QfNaFl_73xCG55FPVCmykQnfPsSPBNIWHdoho.pem
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:13: DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:24: SSLCertificateFile /var/lib/letsencrypt/_lqxWpvqq2aRRJGvb5fw8f6MTKtKjvaOQG63G1CdZT8.crt
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:25: SSLCertificateKeyFile /var/lib/letsencrypt/_lqxWpvqq2aRRJGvb5fw8f6MTKtKjvaOQG63G1CdZT8.pem
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:27: DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:38: SSLCertificateFile /var/lib/letsencrypt/zjhpMNyXvkjp14pCN87KIKPcajcrHVSOgSHLEPSsoJA.crt
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:39: SSLCertificateKeyFile /var/lib/letsencrypt/zjhpMNyXvkjp14pCN87KIKPcajcrHVSOgSHLEPSsoJA.pem
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:41: DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:52: SSLCertificateFile /var/lib/letsencrypt/xV1bGGq4CWl869V6GGZToEOjKAXH20Glffes4tOE7as.crt
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:53: SSLCertificateKeyFile /var/lib/letsencrypt/xV1bGGq4CWl869V6GGZToEOjKAXH20Glffes4tOE7as.pem
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:55: DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:66: SSLCertificateFile /var/lib/letsencrypt/QMiFyo1FQGJNQtUr1CVEiKLksSWRWF0MOYTdi4XBYQ4.crt
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:67: SSLCertificateKeyFile /var/lib/letsencrypt/QMiFyo1FQGJNQtUr1CVEiKLksSWRWF0MOYTdi4XBYQ4.pem
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:69: DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:80: SSLCertificateFile /var/lib/letsencrypt/sv2cP5DqvGlVJmgyw91JBfVMVDWgpHhLPnADdP03704.crt
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:81: SSLCertificateKeyFile /var/lib/letsencrypt/sv2cP5DqvGlVJmgyw91JBfVMVDWgpHhLPnADdP03704.pem
/etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:83: DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
I commented out the lines that refer to /var/lib/letsencrypt and ran certbot renew again. This created new crt and pem files in /var/lib/letsencrypt:
ls -la /var/lib/letsencrypt
total 64
drwxr-xr-x 4 root root 4096 Nov 20 14:56 .
drwxr-xr-x. 32 root root 4096 Nov 18 14:12 …
drwxr-xr-x 2 root root 4096 Nov 18 14:12 backups
-rw-r–r-- 1 root root 1147 Nov 20 14:56 _lqxWpvqq2aRRJGvb5fw8f6MTKtKjvaOQG63G1CdZT8.crt
-r-------- 1 root root 1704 Nov 20 14:56 _lqxWpvqq2aRRJGvb5fw8f6MTKtKjvaOQG63G1CdZT8.pem
-rw-r–r-- 1 root root 1147 Nov 20 14:56 QMiFyo1FQGJNQtUr1CVEiKLksSWRWF0MOYTdi4XBYQ4.crt
-r-------- 1 root root 1704 Nov 20 14:56 QMiFyo1FQGJNQtUr1CVEiKLksSWRWF0MOYTdi4XBYQ4.pem
-rw-r–r-- 1 root root 1147 Nov 20 14:56 spv3X8QfNaFl_73xCG55FPVCmykQnfPsSPBNIWHdoho.crt
-r-------- 1 root root 1708 Nov 20 14:56 spv3X8QfNaFl_73xCG55FPVCmykQnfPsSPBNIWHdoho.pem
-rw-r–r-- 1 root root 1147 Nov 20 14:56 sv2cP5DqvGlVJmgyw91JBfVMVDWgpHhLPnADdP03704.crt
-r-------- 1 root root 1704 Nov 20 14:56 sv2cP5DqvGlVJmgyw91JBfVMVDWgpHhLPnADdP03704.pem
drwxr-xr-x 2 root root 4096 Nov 20 14:56 temp_checkpoint
-rw-r–r-- 1 root root 1147 Nov 20 14:56 xV1bGGq4CWl869V6GGZToEOjKAXH20Glffes4tOE7as.crt
-r-------- 1 root root 1704 Nov 20 14:56 xV1bGGq4CWl869V6GGZToEOjKAXH20Glffes4tOE7as.pem
-rw-r–r-- 1 root root 1147 Nov 20 14:56 zjhpMNyXvkjp14pCN87KIKPcajcrHVSOgSHLEPSsoJA.crt
-r-------- 1 root root 1704 Nov 20 14:56 zjhpMNyXvkjp14pCN87KIKPcajcrHVSOgSHLEPSsoJA.pem
and created a new /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf file which refers to them.
The renewal failed with the same error message I got when the initial renewal failed on Nov 15:
sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log
letsencrypt_log.txt (38.9 KB)
Ok, we’re progressing 
What are the permissions of /etc/httpd/httpd.conf? You can check this with ls -l /etc/httpd/httpd.conf
This is the situation I was looking to get audit.log entries for previously, in Renewal Failure Centos 7 (comment). If httpd.conf permissions don’t give explanation for this behavior, we might have to repeat that.
To do that, we’ll first have to reset the state again. Btw. /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf can be safely deleted. But let’s first see if simply httpd.conf permissions would be the core problem.
We are indeed progressing. In-between conversing with you, I emailed my brother-in-law. When I mentioned that you asked about the permissions of httpd.conf, he asked me about the immutable bit. Sure enough, lsattr /etc/httpd/conf/httpd.conf showed:
----i--------e-- /etc/httpd/conf/httpd.conf
After clearing this bit, the certs renewed.
He had a few choice words to say to me which I won’t repeat here. This is a lesson for sure. You have been unbelievably patient, and I cannot thank you enough.
Charlie
No, thanks to you, and sorry about the inconvenience. With your help, we were able to spot two UX affecting bugs in our code. In no situation should Certbot get into situation that it can’t recover from. Especially between runs.
The bugs were:
- Writing file change info to temporary checkpoint regardless of the file being actually changed
- Not being able to recover from a situation where temporary checkpoint info is out-of-sync with actual filesystem.
So this actually improved the software quite a bit!
1 Like
I guess one could then say that this thread was not just idle chattr.
2 Likes
Your kindness is appreciated! I am glad to have been of assistance if in a somewhat left-handed way (I am left-handed, so perhaps this is appropriate).
1 Like
I agree, but who is going to have the last lsattr.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.