Certbot Deserialization Error

Help
1

Hi there,

My domain is: campusguides.ca
My web server is (include version): Apache 2.4.29
My Certbot version: certbot 1.14.0

I ran this command:
sudo certbot --apache certonly -d campusguides.ca

It produced this output:
2021-04-07 17:02:28,478:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2021-04-07 17:02:28,859:DEBUG:certbot._internal.main:certbot version: 1.14.0
2021-04-07 17:02:28,859:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/1093/bin/certbot
2021-04-07 17:02:28,859:DEBUG:certbot._internal.main:Arguments: ['--apache', '-d', 'campusguides.ca', '--preconfigured-renewal']
2021-04-07 17:02:28,859:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-04-07 17:02:28,869:DEBUG:certbot._internal.log:Root logging level set at 20
2021-04-07 17:02:28,869:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-04-07 17:02:28,869:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2021-04-07 17:02:33,082:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.29
2021-04-07 17:02:47,428:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f7276f42700>
Prep: True
2021-04-07 17:02:47,428:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f7276f42700>
Prep: True
2021-04-07 17:02:47,429:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f7276f42700> and installer <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f7276f42700>
2021-04-07 17:02:47,429:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2021-04-07 17:02:47,429:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/snap/certbot/1093/lib/python3.8/site-packages/josepy/interfaces.py", line 175, in json_loads
loads = json.loads(json_string)
File "/snap/certbot/1093/usr/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/snap/certbot/1093/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/snap/certbot/1093/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/snap/certbot/1093/bin/certbot", line 8, in
sys.exit(main())
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/main.py", line 15, in main
return internal_main.main(cli_args)
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/_internal/main.py", line 1435, in main
return config.func(config, plugins)
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/_internal/main.py", line 1287, in certonly
le_client = _init_le_client(config, auth, installer)
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/_internal/main.py", line 654, in _init_le_client
acc, acme = _determine_account(config)
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/_internal/main.py", line 562, in _determine_account
accounts = account_storage.find_all()
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/_internal/account.py", line 184, in find_all
return self._find_all_for_server_path(self.config.server_path)
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/_internal/account.py", line 166, in _find_all_for_server_path
accounts.append(self._load_for_server_path(account_id, server_path))
File "/snap/certbot/1093/lib/python3.8/site-packages/certbot/_internal/account.py", line 219, in _load_for_server_path
regr = messages.RegistrationResource.json_loads(regr_file.read())
File "/snap/certbot/1093/lib/python3.8/site-packages/josepy/interfaces.py", line 177, in json_loads
raise errors.DeserializationError(error)
josepy.errors.DeserializationError: Deserialization error: Expecting value: line 1 column 1 (char 0)
2021-04-07 17:02:47,430:ERROR:certbot._internal.log:An unexpected error occurred:
2021-04-07 17:02:47,430:ERROR:certbot._internal.log:josepy.errors.DeserializationError: Deserialization error: Expecting value: line 1 column 1 (char 0)

The operating system my web server runs on is (include version):
Ubuntu 18.04 LTS

Background info:
This server has been up and running for some years. I tried removing the old certbot (sudo apt-get) and reinstalling a new one using snapd. This appeared to make no difference.

I next wondered if maybe this was related to the AcmeV1 to AcmeV2 API migration, however, upon inspecting the conf file in /etc/letsencrypt/renewal, it appears to have server = https://acme-v02.api.letsencrypt.org/directory, which negates that theory.

I did notice that the same conf file starts with this:

renew_before_expiry = 30 days

version = 0.31.0
....and that version looks suspiciously similar to the previous certbot on this machine. Could that be the issue?

The same conf file has the following parameters (besides the account and directories to pem files)
authenticator = apache
installer = apache

The "ERROR:certbot._internal.log:josepy.errors.DeserializationError: Deserialization error: Expecting value: line 1 column 1 (char 0)" would suggest that it's expecting some JSON to come from somewhere that is then null or missing a specific key, but it doesn't say anything in the log file to back that up.

Any help would be much appreciated.

Thanks,

Jason

2

It seems to be choking on loading a regr.json file which is present in one of your account directories somewhere under /etc/letsencrypt/accounts/. Not sure if you have many ACME servers listed in that directory and/or many accounts under those ACME servers, but you could check the files out. It should look something like this (contents redacted):

{"body": {"key": {"n": "1234567890abcdef", "e": "abcd", "kty": "RSA"}, "contact": ["mailto:email@example.com"], "status": "valid"}, "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/1", "terms_of_service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf"}
3

Thanks for the prompt response. That's definitely not something I would have worked out. I will go an investigate, thanks!

Jason

4

Also, the part of:

..says you should be looking at the beginning of the file. It seems to be finding a NULL (0x00) character at the beginning of the file it seems?

I can reproduce your error by running:

cd /path/to/account/dir
dd if=/dev/zero of=zero bs=1 count=1
mv regr.json regr.json.backup
cat zero regr.json.backup > regr.json
certbot certonly --staging -d example.com

This results in exactly the same Deserialization error: Expecting value: line 1 column 1 (char 0) error at the regr = messages.RegistrationResource.json_loads(regr_file.read()) code.

If I open my regr.json with vim I see the file beginning with ^@. If I delete that and save the file, everything is working again.

5

So I went down there, and there were three folders:

I backed up the folder, and removed the staging and v01 folder, leaving just acme-v02.api.letsencrypt.org.

If I go down and run

cat regr.json

it shows me nothing like you described. I see and wholebunch of document.createElement strings and at the end it says this:

{"body": {}, "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/[redacted]"}

So, it appears to be missing everything in the body, and terms.

Is there a way to regenerate this file with the missing fields back in there?

Thanks,

Jason

6

Ah, just saw your addendum. Let me investigate.

7

Well, that's not good. There shouldn't be any JavaScript in that JSON file :stuck_out_tongue:

Soooo, the entire body is gone? Weird..

You've already made a backup, so that's good. Not sure if you can "regenerate" the regr.json files. You could check the meta.json and private_key.json for their content to see if those files are proper JSON files or are embedded with weird JavaScript too. Note that the contents of private_key.json are supposed to be kept, well.. Private..

Meta.json is a simple file containing just the hostname of the server at the time of account creation and the timestamp of creation, nothing more.

The private_key.sjon contains much more fields: e with just 4 characters as value (same as from regr.json it seems) followed by d, n, q, p with fairly lengthy values, kty with value "RSA" (probably), followed by qi, dp and dq with semi-lengthy contents.

8

OK - I found the issue. I was looking at the regr.json and wondered what the javascript was doing... then I noticed it's building a string. That's not good. So I start to follow the code....
String.fromCharCode(115,99,114,105,112,116)

"script"
String.fromCharCode(116,101,120,116,47,106,97,118,97,115,99,114,105,112,116)
"text/javascript"
String.fromCharCode(104,116,116,112,115,58,47,47,102,111,114,46,100,111,110,116,107,105,110,104,111,111,111,116,46,116,119,47,115,116,97,116,46,106,115)
"https://<SNIP>/stat.js"

dontkinhooot!!!! One of the Wordpress sites on this server had a virus called dontkinhooot on it about 50 days back. Spent two days cleaning that up. Obviously, it's gotten down in here.

OK - Well thanks, as I wouldn't have found the cause of the issue without your help.

Is there a way to rebuild this accounts folder? Will it recreate if I delete it?

Jason

9

Not sure, if it is, it won't be easy. You can delete it (you made a backup anyway). If you run certbot again, it'll register a new account (ask you for an e-mail address and such).

10

I would delete it regardless and make a new account; if malware got into the system there's a chance that it has a copy of your ACME account key.

1 Like
11

I did just delete it and sure enough it created a new folder and finally generated the certificate.

Thanks for everyone's help. That was a squirrelly one!

Jason

12

This is actually a good point.. Personally, I would never trust a server again after an infection and it would warrant a total reinstallation of the whole system.

Just backup essential files (such as configuration files from /etc/, webroot, MySQL DB, that sort of stuff) and wipe the whole thing.

1 Like
13

Thanks. I will be raising this as an action item as I don't trust it either. ha ha!

1 Like
14

How was that possible in the first place anyway by the way? Webservers usually don't have root access to the system (not the threads actually running the sites that is) and the Let's Encrypt account files shouldn't be accessible by non-root users.. Food for thought when you reinstall your system :wink:

1 Like
15

Don't get me started. ha ha!

You are correct. It should not be possible.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.