OCSP responder Problems? (letsencrypt.org DNS Problem)

futureweb · July 30, 2018, 6:20pm

Hey there,
are there currently Problems with OCSP?
My Apache Logs getting spammed with:

[Mon Jul 30 20:18:00.129016 2018] [ssl:error] [pid 20230:tid 140385049319168] (EAI 2)Name or service not known: [client 217.88.235.x:58036] AH01972: could not resolve address of OCSP responder ocsp.int-x3.letsencrypt.org
[Mon Jul 30 20:18:00.129041 2018] [ssl:error] [pid 20230:tid 140385049319168] AH01941: stapling_renew_response: responder error
[Mon Jul 30 20:18:00.139422 2018] [ssl:error] [pid 3514:tid 140385116460800] (EAI 2)Name or service not known: [client 87.162.251.x:58397] AH01972: could not resolve address of OCSP responder ocsp.int-x3.letsencrypt.org
...
...
...
[Mon Jul 30 20:18:02.174108 2018] [ssl:error] [pid 20854:tid 140385150031616] (EAI 2)Name or service not known: [client 31.150.210.x:56004] AH01972: could not resolve address of OCSP responder ocsp.int-x3.letsencrypt.org

And Websites are not reachable? (Example: https://www.die-glaserei-parger.at/)

thx, bye from Austria
Andreas Schnederle-Wagner

ps) @cpu - as I don't know whom else to ping here ...

stevenzhu · July 30, 2018, 6:29pm

Hi,

Can you please try an ping to the above address?

ping ocsp.int-x3.letsencrypt.org

Thank you

futureweb · July 30, 2018, 6:32pm

Server:
# ping ocsp.int-x3.letsencrypt.org
ping: ocsp.int-x3.letsencrypt.org: Name or service not known

Local PC:
C:\Users\hellkeeper>ping ocsp.int-x3.letsencrypt.org
Ping-Anforderung konnte Host “ocsp.int-x3.letsencrypt.org” nicht finden. Überprüfen Sie den Namen, und versuchen Sie es erneut.

Both located in AUSTRIA (Server = Vienna, PC = Tirol)

cpu · July 30, 2018, 6:33pm

This is being investigated. There will be a status update shortly

futureweb · July 30, 2018, 6:35pm

# dig +trace ocsp.int-x3.letsencrypt.org

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> +trace ocsp.int-x3.letsencrypt.org
;; global options: +cmd
.                       496487  IN      NS      k.root-servers.net.
.                       496487  IN      NS      c.root-servers.net.
.                       496487  IN      NS      h.root-servers.net.
.                       496487  IN      NS      j.root-servers.net.
.                       496487  IN      NS      l.root-servers.net.
.                       496487  IN      NS      f.root-servers.net.
.                       496487  IN      NS      g.root-servers.net.
.                       496487  IN      NS      b.root-servers.net.
.                       496487  IN      NS      i.root-servers.net.
.                       496487  IN      NS      d.root-servers.net.
.                       496487  IN      NS      m.root-servers.net.
.                       496487  IN      NS      a.root-servers.net.
.                       496487  IN      NS      e.root-servers.net.
.                       517938  IN      RRSIG   NS 8 0 518400 20180812170000 20180730160000 41656 . RsNN1xtIHOdLKEGpm0Z1cRO4F6KvrfQJwWbi9usrwgd4vO8SMKf2FL3+ a8OPz6QtPrzKQaRVXtE/oUdT5baYAWLbXQAFJ1hjcc/qHcvWBlQngu1C 53UoCGkDw4RWQB6Pn++6kJ+STLXZQG3nbOwyd7aG86Aixw1KreulcNwh XIo7y5AokW2LZ4Y2VxsYJdPWovDRwkJHmQmhVVdi1Eq+7Jt8t9V+mt2M utGyr6ZOnb+0favrzlaXyl8wgU3V6etd54JxxzzBHZrEgSk5lFwzvpAR fKAKN4xkluB7VJ5AjzQdXzw/Jp4GiGwUNmIfzW3g3uxhj5+rUfX/ul2r eZTlVA==
;; Received 1097 bytes from 83.64.177.250#53(83.64.177.250) in 7 ms

org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      d0.org.afilias-nst.org.
org.                    86400   IN      DS      9795 7 2 3922B31B6F3A4EA92B19EB7B52120F031FD8E05FF0B03BAFCF9F891B FE7FF8E5
org.                    86400   IN      DS      9795 7 1 364DFAB3DAF254CAB477B5675B10766DDAA24982
org.                    86400   IN      RRSIG   DS 8 1 86400 20180812170000 20180730160000 41656 . joqnfkfl7RoB1MIv/4uTszz7C+BxMLittiFBICo3+1n0RwvL/vFm545H 1sWbtSYFfOlpRgx9Z7OfboOaOl4uIHlo5yoj2tDgx4pC8HT3Ix4ZxF0w z+ZKLzIrQ2efqZDgtrmWCG0OauOVLW8rCA3qb2zkhykMN4d/eXTQ8ZR5 /ymPIHGYJvK1VSxHFvfWR9eA97bxBByTodqmTFP/KOPTDa3D4wxKnZaC 0UZWvyhUGngMwiCiejRXzuzqpph9fw5TspSIPFcItogC7qoInSuvmA0s BGA2O5GFgW1Qk601mzXxmBCaJI+SpiD17SiGipFDFF2KdGszIihSOika LFuxyQ==
;; Received 829 bytes from 192.112.36.4#53(g.root-servers.net) in 47 ms

org.                    900     IN      SOA     a0.org.afilias-nst.info. noc.afilias-nst.info. 2013087797 1800 900 604800 86400
org.                    900     IN      RRSIG   SOA 7 1 900 20180820183418 20180730173418 1862 org. L8AVeXWXYg5G7pHlRYQOOVuWrbTFhZ7THI6DUbpMXwOAUGJ7hwiuUEZl w+gDMw+7HnSVnZEB0sQtPU/OqXYWJuKXV+isgHV/bJLUn1pz4DIpHj2O DX/gr2K9Z1tnvA1fRx+fmHIiMNKsMqgSMjmWAB6SOX7BsH0XGIHYg7Z3 BLY=
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN NSEC3 1 1 1 D399EAAB H9PARR669T6U8O1GSG9E1LMITK4DEM0T NS SOA RRSIG DNSKEY NSEC3PARAM
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN RRSIG NSEC3 7 2 86400 20180820183418 20180730173418 1862 org. SS+evUW6LUSXyP93Rh0IirVqkdOffIqgr3xCRAF+bTP9VWBEFy6wyjBR zTXubuYb88GWvdcMum5NqCSJPjB7jbztjixNqZnbtjIf0o4wuLyLRBcS dnGsRFOwCHF0yyP7xWw/uqv5Opj/pCzdivMFJ/7NvlRmj5KdVPCQdPe5 bOY=
6tudcfrknr572i5c0uc4sacr7a29acu9.org. 86400 IN NSEC3 1 1 1 D399EAAB 6TULFBOA2428TKUTFU787KBL9URP7H08 NS DS RRSIG
6tudcfrknr572i5c0uc4sacr7a29acu9.org. 86400 IN RRSIG NSEC3 7 2 86400 20180815153044 20180725143044 1862 org. I+3ZnpqwwR/WN79jBybW7r8YBUy63A0VTylmOino6SlioFfyItUTsuqa fQGrKJXr+uUijS1zSupl5GpN1Q67ndC643cH1X/DLuQ8Y1Ls6gDDv/hR kI6YWTKt7WRblVcd7zMX9uVGth38xk2Olox2et+CP/TsXs7qEr17/yyY Uuw=
vait3kqm4ddfgbgsl5ch2adeqcdaekt4.org. 86400 IN NSEC3 1 1 1 D399EAAB VAJB898DELVT5UJ4I9D1BRD2FRTBSCM1 NS DS RRSIG
vait3kqm4ddfgbgsl5ch2adeqcdaekt4.org. 86400 IN RRSIG NSEC3 7 2 86400 20180815153044 20180725143044 1862 org. CI6sSDeRe8zmbG5pqVwKkfduFmfGczbyMvVEGCXkN6TqG9+dD9qGjz9w RuwdKwV1XFk/+06a2FlsdhtOf8TSAu2uW0E4SwAOoaKof53APp+RD5xU 6A17pWrT6f73cJPzaTf1EUxUiDXNexelyvPgb/WpHEQ9GzRumMYYGOx8 wzg=
;; Received 1021 bytes from 199.19.57.1#53(d0.org.afilias-nst.org) in 155 ms

kepstin · July 30, 2018, 6:37pm

Yeah, it looks like there’s an issue with the DNS resolution of the letsencrypt.org domain. The glue records are missing from the DNS servers hosting the org. TLD, so recursive resolvers can’t find letsencrypt.org

This is affecting services other than OCSP, I can’t renew certificates either right now.

(This also is preventing people from accessing community.letsencrypt.org unless they happen to still have it in their dns cache!)

Phil · July 30, 2018, 6:57pm

Kepstin,

That is correct. We’re busy reaching out to our registrar to determine why this happened. We’ll update the status.io more once we have details.

sahsanu · July 30, 2018, 7:52pm

Chaising namecheap…

https://twitter.com/0xjosh/status/1024009279152508928

futureweb · July 30, 2018, 8:10pm

seems like the chase was successful …

futureweb · July 31, 2018, 7:32am

Seems like the Problem is solved? On my DNS Servers Name Resolution is working again for letsencrypt.org …
Not sure if it’s already replicated worldwide …
Going to activate OCSP again on our Servers …

Andreas

ps) looking forward to incident report what went wrong here … (don’t tell me it was a simple “forgot to renew/pay Domain issue”?!? )

sahsanu · July 31, 2018, 7:37am

I don't think so

$ whois -h whois.enom.com letsencrypt.org | grep Date
Updated Date: 2015-06-08T04:35:04.00Z
Creation Date: 2014-07-07T19:54:00.00Z
Registrar Registration Expiration Date: 2025-07-07T19:54:04.00Z

futureweb · July 31, 2018, 7:39am

haha - ok
just saw a reference to “clientHold status” of the domain on https://letsencrypt.status.io/

tomwas · August 1, 2018, 9:49am

Will a post mortem analysis be published about this outage? Considering the scale of the outage (not just OCSP), I think many people would appreciate this.

cpu · August 3, 2018, 1:18pm

A postmortem has been published: 2018.07.30 Domain Resolution Interruption

system · September 2, 2018, 1:18pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.