Certbot 5.0.0 - Unable to refresh certificate - request message was malformed :: Content-Type must be "application/jose+json"

Hey everyone, hope this message finds you well! :blush:

After having run my home server for a few months successfully using Let's Encrypt in combination with Certbot, I'd first of all want to thank you for all your efforts; it truly made switching to SSL certificates incredibly easy!!

Now, today, my certificates would have been due for removal, and it seems like the renewal process via certbot failed. I'll use your template to further describe this issue.

If you need anything else (e.g. the (full) logfile[s]), please let me know :blush:

But as a layman looking in from the outside, I'd assume that the most interesting pointer would indeed be the Content-Type given for "POST /acme/new-order HTTP/2.0", as indicated from this section within the log:

2025-09-25 11:32:26,574:DEBUG:acme.client:Received response:
HTTP 400
server: nginx
date: Thu, 25 Sep 2025 09:32:26 GMT
content-type: application/problem+json
content-length: 194
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: qRkKkMTFyzmKT0qGoob8ZlXER7hAX2gE_4t42jOScfyU3m4oruQ

{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Unable to validate JWS :: Invalid Content-Type header on POST. Content-Type must be \"application/jose+json\"",
  "status": 400
}

Well, this, and the weird Apache version mismatch (see below: "It produced this output")


My domain is:
bmn.ddns.net

I ran this command:
sudo /home/[me]/bin/miniforge3/envs/core/bin/certbot renew

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/bmn.ddns.net.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Unable to read ssl_module file; not disabling session tickets.
Certbot has detected that apache version < 2.4.11 or compiled against openssl < 1.0.2l. Since these are deprecated, the configuration file being installed at /etc/letsencrypt/options-ssl-apache.conf will not receive future updates. To get the latest configuration version, update apache.
Renewing an existing certificate for bmn.ddns.net
Failed to renew certificate bmn.ddns.net with error: urn:ietf:params:acme:error:malformed :: The request message was malformed :: Unable to validate JWS :: Invalid Content-Type header on POST. Content-Type must be "application/jose+json"

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/bmn.ddns.net/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
Apache/2.4.65 (Linux/SUSE)
Build timestamp: 2025-09-23

The operating system my web server runs on is (include version):
openSUSE Leap 15.6 (patched up)

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no (all via ssh / cockpit)

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
5.0.0 (via conda-forge)

Can you also show what Certbot actually send? Above this "received response" there should be a payload message being send.

Oh yeah, sure, thank you so much for your help! :blush:

Here you go (I was just not sure if I'd share security-relevant info if I included the full logfile, that's why I was waiting for a specific question regarding this. I decided to omit the actual values of this request, but if they are important, I'll happily edit them in. Let me know if that's the case!):

2025-09-25 11:32:26,415:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "bmn.ddns.net"\n    }\n  ]\n}'
2025-09-25 11:32:26,419:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "[...]",
  "signature": "[...]",
  "payload": "[...]"
}
2025-09-25 11:32:26,573:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/2.0" 400 194
2025-09-25 11:32:26,574:DEBUG:acme.client:Received response:
[...]

Well, the weird thing is:

The code responsible for the POST-ing is:

And that function uses the required application/jose+json content type through JOSE_CONTENT_TYPE. And that code hasn't been touched in YEARS.

Also, with my Certbot I'm not getting any error, which suggests there's also nothing weird going on at the ACME server side of things?

Hmm, unless the content_type argument for _post_once() is overridden somewhere else. Let me dive deeper. Or actually, more shallow :stuck_out_tongue:

Nope, the _post() call for the new order:

doesn't specify any specific content type header..

And all the functions in between don't modify the arguments.

2 Likes

Hm, I might try setting up a remote debugger to my server and set a breakpoint on _post_once maybe? I'd let you know if the parameter is what is to be expected or might - for some weird reason - actually differ?

Or is there a more "streamlined" way built-in into certbot, to troubleshoot cases like these? :sweat_smile:

I dunno, maybe add a debugging print/log statement which actually prints the headers being send :man_shrugging: To see what's actually being send over the wire. Breakpoints should also work ofc.

By the way, I tested on the staging server. Although that should resemble the production server and it's highly unlikely that the production server would run newer code than production, this is something to keep in mind.. There could be configuration differences, although I don't think it's likely these kinds of things would be a configuration item.

1 Like

Welp, yeah, this is discouraging:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/bmn.ddns.net.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Unable to read ssl_module file; not disabling session tickets.
Certbot has detected that apache version < 2.4.11 or compiled against openssl < 1.0.2l. Since these are deprecated, the configuration file being installed at /etc/letsencrypt/options-ssl-apache.conf will not receive future updates. To get the latest configuration version, update apache.
Renewing an existing certificate for bmn.ddns.net
Using content-type: application/jose+json
Failed to renew certificate bmn.ddns.net with error: urn:ietf:params:acme:error:malformed :: The request message was malformed :: Unable to validate JWS :: Invalid Content-Type header on POST. Content-Type must be "application/jose+json"

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/bmn.ddns.net/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Anyway, thank you so much for taking your time to look into this! :sweat_smile:

Maybe it is related to the weird Apache version thing? Is this a known issue? Could this possibly be related? What I mean is this error here:

Certbot has detected that apache version < 2.4.11 or compiled against openssl < 1.0.2l. Since these are deprecated, the configuration file being installed at /etc/letsencrypt/options-ssl-apache.conf will not receive future updates. To get the latest configuration version, update apache.

while my Apache version is actually higher (Apache/2.4.65), and my openssl as well, of course (OpenSSL 3.5.2 5 Aug 2025 (Library: OpenSSL 3.5.2 5 Aug 2025).

I mean, again, from the outside looking in, I couldn't imagine how this could be connected, but at this point, I'd have no idea what else to contribute here :melting_face:

I don't think this is related. The apache installer/authenticator Certbot plugin is not related to the acme library that's responsible for the ACME client part of the code.

What ACME server is Certbot connecting to? Is there a proxy in between?

1 Like

Well, at this point I might need a little bit more handholding, since networking really isn't my strong suite at all :woozy_face:

Is this output useful at all, to you?

> nslookup acme-v02.api.letsencrypt.org
Server:         212.186.211.21
Address:        212.186.211.21#53

Non-authoritative answer:
acme-v02.api.letsencrypt.org    canonical name = prod.api.letsencrypt.org.
prod.api.letsencrypt.org        canonical name = ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
Name:   ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com
Address: 172.65.32.248
Name:   ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com
Address: 2606:4700:60:0:f53d:5624:85c7:3a2c

Also, I did not configure any kind of proxy, from what I know, no! Only proxies involved are apache2 for proxying/reverse-proxying my local services, but I just made double sure that there really is none in there pointing to letsencrypt.org or the like!

Hmkay, then I'm out of ideas :man_shrugging:

Maybe some other volunteer has any good suggestions.

1 Like

Aw, I understand, I fully appreciate your help and patience though, thank you so so much!!

2 Likes

That is a known issue but supposedly was fixed in V5 which you already have. See the EFF's Github for Certbot: certbot/certbot/CHANGELOG.md at main ยท certbot/certbot ยท GitHub

You can safely ignore that message.

And, I agree with osiris that it is not related to the content-type error.

So, was that a pip venv install?

Did the content-type error just begin when you upgraded to 5.0 ? I see you got the first cert for your domain in April so must have used an earlier version for that.

As an aside, have you considered using Apache's built-in ACME Client mod_md. The Certbot problem may well turn out to be a packaging or incompatibility with your system. If you don't need a stand-alone ACME Client mod_md may be easier. See: mod_md - Apache HTTP Server Version 2.4

Certbot's --apache plugin uses same config as Mozilla's configurator here: Mozilla SSL Configuration Generator
Set your Apache / openssl versions

4 Likes

Yes, exactly! Afaik, conda/mamba/... create a managed venv environment (and a separate packaging repository). I was just in the middle of figuring out an alternative way to install certbot, given I noticed that my certbot-apache client was behind the official pypi version.

But this sounds even better than troubleshooting my python environment, in this case. I'd rather avoid snap, at least for now, so this does sound like a viable option to explore (and I agree, this could very well be the issue), hopefully this somehow solves it!

Super dumb question from my side now: uhm, is there an alternative to the system's package manager / repos for installing apache2 modules? Totally understand if this isn't the place to ask for this, though!

Honestly, I am not 100% sure. I set up my systemd timer unit for certbot, and that was that. This is the first time since that very moment, this is occurring. So, true, the renewal in June must have worked, and given the June certificate expired today... yeah! But I upgraded to 5.0.0 after encountering this issue this morning! So, I was running the previous version before, where it was occurring too.

I'm not sure I understand the question / concern. How did you install Apache itself?

2 Likes

Yeah, I understand your confusion - I thought the mod_md module was packaged separately (within my OS's packages), but it turned out that isn't the case, since I successfully enabled it, without having to install any additional packages.

I'm now just fighting with my apache2 instance's configuration. I'll report back as soon as I know if this worked.

2 Likes

Awesome! Everything is up and running again, thank you so much @MikeMcQ :blush:

Switching to mod_md worked really well (after some meaningful discussions with my apache2 configuration), and, for future references, following this specific guide:

The apache2 documentation of this module itself is a great reference, but omits one or two rather details regarding the usage context of letsencrypt (MDCertificateAgreement accepted, ServerAdmin needs to be configured).

I'll mark your response as the solution for my specific issue and this topic can thus be closed.

Thank you all for your time and hope you're doing well;
Cheers!

3 Likes

You are very welcome. I usually include the link that you did for that github. I've lately been omitting it when recommending mod_md but I'll be sure to include it in the future - good reminder, thanks :slight_smile:

3 Likes