Certificate error with large SAN cert


#1

I am trying to extend a SAN cert, as I have done multiple times in the past, yet this time I am receiving an error:

{
“type”: “urn:acme:error:serverInternal”,
“detail”: “Error creating new cert”,
“status”: 500
}

I checked the server status page, as well as the history and there does not seem to be any current issues with the le servers. Our server is running the current version of certbot (certbot 0.17.0) on an ubuntu 14.04 server that is hosting content via nginx.

As an aside, I accidentally requested a cert in the process of testing with only 2 hostnames included in the SAN field (the current failing request has 97,which I realize is 3 short of the limit) and that succeeded.

I’m running ‘certbot certonly --webroot --expand -w /path/to/webroot -d domain -d domain2 -d domain3 -d domain4 …’

It outputs:
http-01 challenge for domain97
Using the webroot path /path/to/webroot for all unmatched domains.
Waiting for verification…
Cleaning up challenges
An unexpected error occurred:
The server experienced an internal error :: Error creating new cert
Please see the logfiles in /var/log/letsencrypt for more details.

Any help would be greatly appreciated.


500s when issuing 8 different certs
#2

Hi @bkc,

What is your ACME account ID?


#3

@cpu its https://acme-v01.api.letsencrypt.org/acme/reg/1798735


#4

Hi @bkc,

It looks like we have some latency problems with issuance for certificates with lots of SANs. We’re working out the best way to solve these in the short term. In the meantime, if putting the new names on a separate certificate is an option for you, that’s probably the best way to get unblocked. Otherwise, we’ll let you know when we’ve deployed a fix.

Thanks,
Jacob


#5

Thanks @jsha, I was just checking on that very thing and trying to find the best way to split these up. Is there a best practice for this? A particular number of SANs/cert I should be under to avoid the latency issues?

FWIW, I was just going to try and split into 2 certs, and then 3 if that didnt work, and so on…


#6

Our goal is to get back to reliably issuing for 100-SAN certificates, so whatever is the easiest and quickest way for you to get past this speedbump is probably best. Your technique of splitting into gradually smaller certificates seems decent!


#7

Hi,

On 55 domains SAN cert we are getting 500 error.I think the CA server’s api is getting timeout after 5 min.

2017-09-27 09:04:04,339:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 “POST /acme/new-cert HTTP/1.1” 500 101
2017-09-27 09:04:04,340:DEBUG:acme.client:Received response:
HTTP 500
Server: nginx
Content-Type: application/problem+json
Content-Length: 101
Boulder-Requester: 21878455
Replay-Nonce: fHxOt4OPfRuCty-2_CfF9_Q4n6_YLQ3gDD4bweEoAe0
Expires: Wed, 27 Sep 2017 09:04:04 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 27 Sep 2017 09:04:04 GMT
Connection: close

{
“type”: “urn:acme:error:serverInternal”,
“detail”: “Error creating new cert”,
“status”: 500
}
2017-09-27 09:04:04,340:DEBUG:acme.client:Storing nonce: fHxOt4OPfRuCty-2_CfF9_Q4n6_YLQ3gDD4bweEoAe0
2017-09-27 09:04:04,341:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.17.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 753, in main
return config.func(config, plugins)
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 692, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 82, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/usr/lib/python2.7/dist-packages/certbot/client.py”, line 357, in obtain_and_enroll_certificate
certr, chain, key, _ = self.obtain_certificate(domains)
File “/usr/lib/python2.7/dist-packages/certbot/client.py”, line 336, in obtain_certificate
domains, csr, authzr=authzr)
File “/usr/lib/python2.7/dist-packages/certbot/client.py”, line 278, in obtain_certificate_from_csr
authzr)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 313, in request_issuance
headers={‘Accept’: content_type})
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 682, in post
return self._post_once(*args, **kwargs)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 695, in _post_once
return self._check_response(response, content_type=content_type)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 582, in _check_response
raise messages.Error.from_json(jobj)
Error: urn:acme:error:serverInternal :: The server experienced an internal error :: Error creating new cert


#8

Our Let’s Encrypt integration service has been receiving 500s when attempting to issue the following 8 certificates:


Www.WatErlInEdAta.COm
cOVeR.CEnTOfante.cOm
WWw.coaLMaRCh.COM
rEstAURANThealTH.cOm
greeNtoWnLABS.COM
WWw.BoISEcHAmBer.org
TEst-lOgZ-IO.lOGZ.io
WaTeRLINeDaTA.coM
lIGhTuPtheMap.orG
BLog.waTeRLineDaTA.coM
gREENPeACe.Org.uK
IgniTEAmERIcA.cOm
WwW.C3iHc.coM
wWW.OnEBAByOwNeR.Co.uk
cusToMTUrfINc.COM
aLlSidEs.com
WwW.reStaUrANThealth.Com
DeVen-ca.oNDecK.COM
C3IHC.COm
uNageNCy.XYz
tRAckItForwaRD.CoM
faNcLuB.HAnKJr.COm
paNTHeOn-sItE-wEB.LOGz.io
WilKHaHn.ch
ONEbabyOWNEr.CO.UK
SeAmonsTErsTuDIOS.coM
LeGEthICs.ny.gov
TesT.PORkBeinspiREd.CoM
WWw.igniTEAmERICA.cOM
LEARNINGpOLIcyINSTITuTE.Org
Www.TrACkITforwaRd.CoM
seCure.crs.orG
acTioNceNteR.Crs.OrG
dONAtE.crS.ORG
www.HOmESpRAyFOAMLlc.Com
gEtLEAdbuIlDer.COm
cONtaineRcHAiN.ch
gwmeDIciNEhEAlTh.Com
Www.c3IsolutioNs.COM
devwww.OndeCK.com
bOiSEcHaMbEr.org
maRKETpLACE.aNImAlSheltErING.oRg
www.CuSTOMtURfInC.coM
WWw.GReenPEAcE.orG.uk
Www.cRANKARMBrEWinG.COm
TEMPLetON.Org
WWW.animAlSHeLTeRINg.oRg
gLpROP.CO.jP
www.mOHAWKcollEgE.cA
wwW.logz.IO
www.tElErx.Com
bRANdCenTer.CRs.ORG
CRSmateriaLs.Crs.oRG
StG.ALlSiDEs.COM
m.GLPRoP.CO.Jp
Www.GwMEDicInehealTH.COm
cHESHiRewIlDLIfeTrust.ORG.uK
WWW.REalLYNEWT.COM
WWwNeXt.OnDECK.cOM
wwWnEXT-cA.ONdeCk.Com
logZ.Io
www.lEarNiNGpoLIcYinSTiTUte.Org
wwW.Afscme17.Org
5752571553644544-Fe2.PANTHEoNsitE.IO
suMadhurAGrOUp.cOm
neWSLEtteRs.cRs.Org
wWW.grEENTOWnlAbS.cOm
www.unAgENCY.Xyz
WWw.CrsmATeRIALs.crS.org
CORp-rESeaRCh.cOM
liVe-LOGz-IO.Logz.IO
c3ISoLuTiONs.cOm
www.seAMoNsTerSTuDIOs.com
coAlmArcH.cOm
wwW.sumaDhuRaGROUp.Com
dev-lOGZ-io.lOgz.IO
rESouRCES.WAterLinEdAtA.coM
getBaCKoFfIcE.COm
www.teMPleton.oRg
aLphA.ENgaGeWISdOM.COM
Www.cOnTAinErchAiN.cH
W.lOGZ.iO
wwW.FLAsHCuTCnC.Com
wWw.WilKhaHN.CH
wwW.LIGhtUpthemap.ORG
coRp-ResEaRCh.ORg
AnImALSHELteriNG.Org
TEST.itHAKA.oRg
WWw.gLPROP.cO.jp
GeTSPROWT.cOm
supPORt.CRS.OrG
Www.IAhollIsWeaLtHtRAnSItIon.cOm
MOHaWKCoLlege.CA
CraNkArmbRewiNG.CoM
WWW.aLlsiDEs.coM
GeTfoRGeLy.cOM
WWw.UspARtNERs.CRs.orG
HomeSpRAyfOamllC.cOm


wWw.DEv.moBIlE.Ibreedit.COm
wWW.MobIlE.IbrEEDIT.cOM
WWw.SANDsAndERson.COM
Www.AzruCo.GOV
DeV.MObiLe.IbReeDit.COM
WWw.jlAURITZEn.CoM
WwW.DIgetT.Com
Www.LILLa-DaN.Dk
ACLu-wy.ORg
WwW.APPS.chApTeRThReE.coM
WWW.TobaCcOCoNTroLREseaRcH.Org
treeSfoRLIfe.ORg.au
WwW.J-lauriTzeN.com
GohEaLthuc.COm
mSJdOMInicanS.orG
wWW.LaurITZENtAnKErS.cOm
cycLETO.cA
lITHOSTaT.cOM
sPSCoMMErCe.Com
wWW.gEtiNKahoOts.COm
WwW.cYcLetO.ca
DEv.EnphasE.cOm
DBE-App.NEt
wWW.TrEESForlife.orG.au
care.oRG
VsPA.berKELeY.eDu
GeTInkahooTs.cOM
WWw.care.oRg
wWW.TrADELINeInc.coM
wWw.laRimerShERIFf.oRG
MarLEymEP.cOM
FlAshtEcHnoLOGy.COM
wwW.liLLADan.DK
araBiANOw.de
AzRuco.Gov
cHINapartner.org
www.dBE-aPp.net
TObACcocONtROlrESEARcH.ORG
eU.huDsONBoATwORks.COm
WwW.csLSHipS.COM
5739719937753088-FE1.pANTheOnSiTE.IO
WWw.WheeLerRENEwaL.bERKelEy.EdU
BLoG.grOuPCARpool.cOM
sanDsaNdeRSOn.CoM
bLoomfiEld.valERetoconnEcT.com
WWw.FLaSHTEChNOLOgY.CoM
wWW.CHiNaPaRTNER.Org
LaRImERSHerIFf.orG
wWW.araBIAnow.De
DUMmYTEStwiSDOMcafE.berkEley.edU
regiSTeR.HEArtfULNESS.ORG
moBiLE.IBreedit.cOM
DigeTt.cOm
wWW.hUdSONBoATWOrKS.coM
WWW.StryVEmARkeTINg.COM
deV.lOnGFELLOwIM.com
WWw.hARlemRBi.ORG
www.LAURItZEnbulkERS.ch
csLShIPs.cOm
aTULoCAlS.Org
GHS.Crs.oRG
WWW.wEAReDReAM.oRg
wWW.cOMMeRcIaLPAYMENtSiNtErnAtiOnal.COm
WWW.mAryville.eDu
camPAIgnS.tRansalT.oRG
Www.LAURItZEnoFfshOresERVIcEs.cOM
BLog.koa.CoM
www.dREamSChOOlNyC.Org
HudSOnBOaTWorKs.Com
www.MARLeymEP.cOM
wWw.LaURiTZEnOFfshOre.Com
WWw.lithOsTAt.com
aDMaReL.EU
tEsT.GOhEALThUc.CoM
sucCEed.CLEArLinK.CoM
wWw.aTUlOcalS.OrG
deV.hAloPETs.CoM
foODfAST.Crs.org
strYVemarketing.COM
dREaMScHOolNyC.Org
acTioN.gROWtHENErgY.orG
commercIAlpayMENtSINterNaTiONaL.COm
WWW.SPSCOMmerCe.COm
Www.JOInMyViLlAGe.coM
wWW.msJDOmiNicaNS.oRG
WWw.pErTherA.cOm
WHEeLERRenEWal.bERkeleY.Edu
weAreDReaM.OrG
PErThERa.Com
WwW.aCLU-wY.OrG
hARLEMRbI.OrG
tESt2.GroUPcARpool.CoM
www.CromweLlFoundaTIon.ORg.au
appS.chAPtERthRee.coM
GerMaN.cHINaPaRTnER.ORg
WwW.GohEaLThuc.Com
mArYvilLe.eDU
WISdomCAfE.beRkeLey.EdU


wWW.HEALThscoPEINC.coM
wWw.dOenteS-iNovAdOREs.Com
GOPiNATa.coM
WwW.cuprESENTs.orG
TEST.KabTEChUsa.COM
paBcOroOFiNg.cOM
WwW.cOLoradOSHaKeS.ORg
hARPERsrETrEAT.cOm
wWw.GoPinaTA.cOm
wWW.THermtecH.nET
WWw.avEdaPDx.com
wWw.cLoUdSiMPLe.COM
foNdATIONDenTRePRiSemARTELL.COM
SPousebuZZ.CoM
DEv.windOwaNDdoor.COM
RelAPtoPs.org
fclAfeStIvAL.cOM
ShinteriorscAPES.cOM
WWw.rOWaNpriCE.COm
glObAlpOvErtY.StaNFORd.Edu
AVeDApDX.cOm
PragerunIverSiTy.cOm
DEV.eLEnd.cOm
GiNkgo-ViEtnAm.cOm
TEsT.EDUhuB.uS.cOm
Www.FcLAFESTIval.COm
CNDev.WoRLdviSIon.oRG
WWw.rICKHoRgER.COm
wWW.chILdreNslAwCenTER.oRG
www.SPousEBuZz.coM
AMErICAneXpRess.TRAvELLER.coM.aU
WWw.gInKgO-viETnAm.coM
www.unITEdpHiLFoRuM.ORG
LIvE.pabCORoOfinG.CoM
ThERmtECH.net
WWW.ShoP-eAt-SuRF.COm
HEALthSCOpeinC.COm
CUPresenTS.COm
www.hARtz.com
doenTES-inoVAdoRes.cOm
cOLORadosHaKes.org
WwW.PLuMBLIneManAgeMeNT.CoM
5112317826039808-Fe1.PAnTheONsiTe.Io
PATiENt-InNOVAtIOn.com
mOntANaCLImatE.oRG
wwW.colOraDoSHAkeSPEareFeSTival.com
WWW.mONtaNACLImAte.ORG
cArEERs.shoP-eAT-surf.coM
STAging.GRAmeENFOuNDaTiOn.orG
www.ShintErIOrScapeS.Com
remSmARTPhArMACY.cOm
LeasEfrOnTEra.CoM
www.pRAGEruniVersiTy.com
pAbcorOOfINg.PaCcoAST.coM
weB.DEV.NEsI.oRG.nZ
coLORaDOshAKes.CoM
hIghlandROw.RAZziNtERaCTIve.cOM
HaRtZ.COm
WWw.PRaGeru.cOm
DeVElOpEr.ASHleYfURnITuRE.coM
www.COloRAdOSHaKeS.com
WwW.PATienT-iNNOvatION.coM
CUprEsentS.oRg
www.foNdAtioNDEnTrePrIsEMaRTeLl.Com
wWw.hARpErsretREAT.COM
wwW.rELapTops.orG
sHop-EAt-SURf.COM
wwW.PaBCoroOFIng.cOm
uNitEDPHIlfORum.ORG
tEst.deVeLopEr.Dev.OfX.COM
LYnNjoHNSONLoCK.Com
atu627.OrG
wWW.RemsMaRtpHaRmAcy.com
wWW.coLORAdOShAkEsPEARefestIvAl.Org
WWw.LyNnjohnSOnLoCk.COm
EDiT.SPLCENtER.oRg
Www.SaFe-LaBS.cOM
www.LeasEfRONteRA.coM
dEv.dEeson.CO.Uk
plUMBLiNEmANAGemEnt.cOm
saFe-labs.COm
rickhorgeR.COm
wWw.SPLCENtER.ORG
ChildrEnsLAwCENter.org
wWW.CupResENts.COm
www.FlUkePRocEssiNstRumENts.Com
tEST.nAnOsphERe.Us
pRAGErU.COM
sPLCeNTer.orG
fluKEpRocessiNstRUmEnTS.com
cLoUdsImpLE.Com


advanCEwaRNingsYSTeMnYC.orG
lancswT.ORg.UK
warWickshirE-WildLIfe-trUst.org.uK
WWW.WaLTHaM-fOsTERiNG-Dev.cMS.FIrMstep.COm
WWW-UaT.cOnTAinercHAin.co.ZA
RWTWAles.ORg
wkWt.orG.uK
mAsSTEENpREgNANCY.OrG
MitEfCOmpEtItiOn.cOM
AquateChPoOlMAnaGEment.COm
dEv.gOHEaLthuc.COm
HRSIMPle.coM
mtmSa.oRG
wpL.shEpeLl.dEMO-siTe.Ca
woRcsWIlDLIFEtrust.CO.uk
WWW.SOuthRIbBLE-cmS-dev.CmS.FirmStep.COm
WwW.AGERo.cOm
DeVOPSroAdTRIp.com
wwW.MitEFcompEtitIOn.coM
WWw.HEriTagEHarVEStfESTivAl.coM
GREEnLeAfASsEssmEnt.UCDAviS.EDU
rEflECTSySteMS.coM
pEaRlpOINT.ORg
gwEntwILDlIFe.oRg
HeRtSWildlIFeTrUST.oRg.Uk
mitEfcOmPeTiTiOn.ORg
aIRbitClUB.InFo
www.GUTstocuTS.cOm
WWW.RWTWALES.ORG
Www.mITeFCOMpetItIon.oRG
Www.MicRoseRvelTD.Co.Uk
GuTsTocUTS.coM
www.aquaTeCHpOOlmANAgeMent.com
clTc.ucdAvIS.EDu
Www.neuEGAlERie.Org
wwW.AdVAnCEwARNiNgSyStemNyc.org
www.rADNORShIreWIldliFETRuSt.ORg.UK
WWW.FiRmSTEP-DEV.CmS.FiRmsteP.COm
ABilIfYMAiNTenA.Ca
Www.PeArlpOInT.oRg
daiLY.JstOr.oRg
fiRmSTEP-deV.Cms.fiRMsTEp.cOm
mICRoSeRveltD.CO.UK
waBAsHceNTER.WAbaSh.edu
neUegaLERIE.orG
WwW.aFFoRdablemOVingSTOrAge.Com
www.MAssteEnPRegNANcy.org
wWW.WOrCSwildlifeTRuST.co.UK
WiLDLondON.Org.uK
sOUthRIbBlE-CMS-DEV.CmS.fIRMStEP.Com
wWw.REfLECTSystEms.COm
wWw.DESTINATiOnYOga.cO.uk
wWW.wABasHCEnter.wAbash.edu
dev.ENErToR.com
wWW.heRiTAGeHaRveSTfEStivAl.OrG
WWw-UAt.coNtaiNERChain.Co.Nl
MIcrOServE.io
WWW.blicHmANNeNGIneeRing.Com
wWW-UaT.CONtAInerChAin.Co.NZ
WWw.direCtsTARTv.cOM
deV.DevopSROAdtRIp.com
Www.MaDIsoNsqUarePArk.orG
ThEPushBaCK.OrG
LIVe.AdvaNCEwArNIngsySTEmnYC.ORg
5666083260334080-FE3.PantHEonsItE.Io
wWW.hrSIMPLE.COM
DEV.aDVAnCeWaRNiNgsYSTEMnYC.oRg
Www.AIRBItClUB.InFO
wwW.wkwT.ORG.uK
hERiTaGEhArveSTfesTiVAl.ORG
HErITAgeHARvESTFESTIvAl.coM
saFE.FrOnTpOIntSecuRiTYsOlUtIoNs.Com
tRANsmitteR.IEee.orG
WwW.iOs-WIldliFetrusT.orG.uk
PLymOutH-DeV.cmS.fIRmStEp.cOm
raDNorshIrEwIlDLifetRUsT.ORG.uK
MAdISoNSquArepark.orG
wwW.tHEPUSHbAck.org
TeST.MasSTeENPrEGnancY.OrG
wWw.MTmsA.org
afForDABLeMOVINGstoRAgE.COM
sTAGiNG.REATCH.CH
Www.pLYmouTh-DEv.cMS.fiRMSteP.com
wARwIcKshIreWildliFETrUsT.org.uK
wWw.WiLdloNDOn.oRg.Uk
IOs-WiLDliFetRuSt.OrG.UK
dEsTINatIonyOgA.co.UK
WWW.waRWiCKSHIre-wilDLife-trUST.orG.uK
walTham-FoSTERinG-dEV.Cms.fiRMsteP.com
sUpPOrT.ReFleCtSYSteMs.cOM
WwW.WaRwicksHIREWilDLiFetRust.org.uk
www.GWeNTWilDlIFe.ORG
bLichmANnengINeErINg.COm
diREcTSTARtv.cOM
sTAGE.GraYmOnT.Com
Www.mIcROSERve.iO
LIve.thEpUSHback.ORg
TEST.tHEpUSHBacK.Org
Www.hERtsWildLIfeTRuST.Org.UK
WWw.lANCSwt.org.UK


LocoS4tRAvel-COstARiCA.Com
WwW.DruPAlcaMppA.Org
cUStOmEr.cynOSUrE.coM
wwW.gloRYREborN.org
worKouTSPoTS.CoM
CalENDaR.ChaTTaNoogasTate.EdU
wWW.MAuIOfFICe.com
CU.edU
tesT.ALcALHOMe.COM
goBwI.coM
wWW.tHESEBasTIAnvail.Com
wILdWILDweb.Es
WwW.lOcOsFoRtRavel.Com
GloRyReBOrn.ORg
www.oYunITeD.Com
wwW.pwpgROWTheqUitY.coM
LOCos4TRaVel.cOm
lOCosfOrTrAVEl.cOm
wWw.resideNcesattheSEbAStIaN.COm
DeV.lOCOs4TRAVel-COSTARicA.com
DruPALCAmPPa.org
Wpl.mOrNeAuSHEPeLL.deMo-siTE.Ca
www.eveRGrEENmd.OrG
AlPHA-DevelopEr.ElaNfINanciaLseRvicES.Com
WWw.OYUNITeD.OrG
AGAKhAnMUSEuM.Org
aTKINsON.MicROseRvE-Qa.iO
WWw-uat.cONTAiNerchain.COm.AU
PWPgROWTheQuITy.CoM
TeSt-DEveLoPeR.elanfiNanciaLSErviCEs.com
theSebastIanVAil.com
WWW.OyuNIted.nET
WindOwANddOOR.COm
WwW.ADoPTIEpets.nL
AmERicomis.com
GaLAvAntIEr.com
WWW.cYNosUre-LOcaTOr.cOm
eveRgReenmD.Org
bAlancEUs.ORg
wwW.petS.be
wWW.americoMIs.Com
test.sIgNwo.MediA
Www.locos4tRAVEL.Com
wW.GALAvantieR.com
dAVidcCOok.COM
reSIDENCESaTThesEbastiAn.cOm
Www.GALAVANtIeR.coM
DruPal.NpaOnLinE.Org
oyUNItEd.Org
WwwW.GALavAntieR.Com
deV.pcf.oRg
wwW.DIEREnasiEL.be
www-UaT.cOntAINerchAIn.CO.Uk
WwW.BusiNeSsWorkshaWAIi.coM
www.thoUGHtleadeRs.WoRlD
tESt.signwo.hOSTING
sPiCEhuNTER.Com
dEV.ZIrAFfE.COM
WWW.CanCERTrEAtMEnTCHOIcEs.com
GloRYREBorN.cOm
peTS.Be
ecomm.cu.EDU
cHallENgESEatTLE.oRg
5641142922117120-fE3.panTHEonSITE.iO
WWw.GOBwi.cOM
oyuNited.inFO
CynoSure-lOcAtor.Com
qA.GeTOUTofDEbt.orG
dierEnaSIel.be
www.wOrKOuTspots.cOM
buSINeSSworKShawaii.COM
WWw.LoCos4trAvEl-cOstAricA.COm
CHalLENGesEAtTLe.COM
www.drupAl.NPaoNlInE.oRg
wWW.IITRi.org
wWW.oYUnItEd.INfO
WwW.DAVIdccOok.com
BlOG.galaVaNTieR.CoM
oyuNiTed.com
BlOg.SmUde.eDu.in
busineSssoLUTIoNShi.com
Www.BuSINeSSsOLuTionSHi.com
CANceRtReaTmENtchoicEs.COm
Www.bALaNCeus.oRG
MAuioFFice.cOM
AdoptIEpets.nl
www.cHAlleNGeSeaTtLe.cOM
wwW.loCOSfortrAvel-CostarICa.cOm
WwW.wIlDwiLdWeB.Es
LocOsFoRTRaVel-cosTaricA.COm
PantheON.EVErGReeNmd.oRg
WWW.AGAKHAnmuSeum.Org
WwW.ChALlEnGESEaTTle.ORg
cHaTtanoOGaStAte.EDU
www-uAT.CoNtAInErChaiN.cOm
dEv.allSIDES.CoM
WWW.gLoryREBOrn.cOM
OYuNiTEd.NeT
WwW.SpiCEHuNTeR.cOm
Cdn.gALAvAnTIER.COM


Www.bROncOSPitcHiN.CoM.au
UWaYwRC.ca
wWW.LiVEAtEVoLUTION.CoM
cPaaCp-CmE.coM
SIlVEr-PEAK.cOm
5636318331666432-FE2.PAnTHeOnsiTe.IO
VIAplACE.Com
wWW.GEtPANtheoN.com
CarIbederm.com
sCC.udel.eDU
RIdemetrObUS.cOm
www.CAdreWoRKs.orG
wWW.ridEMETROBuS.coM
CArpS-cme.oRg
mEetINGS.TravELpOrtland.cOM
WWW.ArABICaLMasDaR.orG
arabicALmasDaR.ORg
DEV.ipW.CoM
lIVE.gLoBalaCaDEmYcme.com
wWw.MsEg.UDEL.EdU
WwW.ABOUTTHEPAtIent.nET
Fdrc.co.Nz
RESearch.uDEl.EDU
www.scc.udEl.edu
WWw.ScPLD.org
chs.udEL.EDU
www.bMe.UDEL.eDU
soRENSENOuTdOoR.COm
WWW.18foR4.coM
Www-Test.GETpaNTHEOn.com
knEeHIpshoUldEr-cmE.ORG
aPPs.gETPANtheoN.coM
WwW.REsEarch.uDel.edu
WWw.SIlVEr-peAk.cOM
WWW.SoRenSENoUtDOOr.COm
appMAnaGeR.fI.edU
LIveAtEvoLUTiOn.cOm
WWw.MERcysHIps.oRg
abouTTHEPaTiEnt.NET
aNHs.ORG
Www.kidNeY.ORg
MSeG.UdEL.eDU
dEv.dANdrEW.COM
wWw.eQsEEd.cOM
mArketPLaCe.silVer-PeAk.Com
Www.SWIeNT.cOm
WwW.CoherEnce.cO.uK
wHEatoNcOLLege.Edu
WwW.SilvErpEaksystEmS.COM
wWW.AnHs.oRG
StCHarLeSlibRary.oRg
BMe.UdEl.edU
18for4.com
SwIenT.cOm
cLARKFounDATiONdC.oRG
meRcYShiPs.oRG
WWw.fdRc.co.NZ
ArChbAlt.orG
DeV.arcADIAdATA.COm
MedviEwmD.coM
sCPlD.ORg
wWW.geTleVelTen.coM
geTlEvElTEn.COm
HElPdesk.GetPANTHEOn.com
swisc.cOM
Www.chs.UDel.EdU
Ifm.oRg
COaSTalDErm.oRG
WWW.FuNCtiONALmEDiciNe.ORg
fUNcTioNaLmEdICiNe.orG
nKF.lI
wWW.MeDViEWMd.cOM
GETPANThEon.cOm
wWW.HeAlthSCiEncE.ORG
WWW.uWAYwRc.CA
WWw.vIapLACE.Com
wWw.pubLIclIBRARyADvOcaCy.Org
Www.STChARleslIBrArY.org
wwW.wHEatoNcoLLEge.edU
www.iFm.orG
deV.RAIL505.COm
tEsT.arCAdIadAtA.Com
GloBalaCAdEmycMe.COm
wwW.StaRGel.cOm
brONCOspitChIN.cOm.Au
HeAlthSciENCe.ORG
stArGeL.coM
inTL.gEtLeveltEN.cOm
wwW.sWiSC.cOM
Help.gEtPANTheoN.COm
wwW.ARcHBaLT.orG
Www-dEv.geTpAnThEoN.coM
PUBLiclIBraRYaDVOcAcY.Org
WWw.cLarKfouNdatioNDc.org
VArNishchEcK.gEtPANTheoN.coM
KidNey.org
eqSEED.com
cAdReWORKs.ORG
SiLVErpEAKSySTeMs.COM
CoHerENce.cO.Uk


loNGVIEW.aRaBElLASENiORCARE.COm
marimOnINc.cOm
wwW.aMErIcANPosSIBiLitIes.ORG
wwW.ImAGINGtEch.NeT
onlINe.Uga.EdU
nGod.GEtOUtDoorscoLorADo.oRg
wWW.MariMoNinc.COM
REsIstAnCEDASHBOArD.COm
copIOusmaNagEMENT.CoM
WWW.SKiLlfUL.Com
TESt1.EVcco.cOM.AU
HErEToheLP.bc.cA
www.bLoG.mORNEAusHePELL.COm
WWW2.PRecept.oRg
wWW.PuC-mpls.org
wWw.linguistIcsOCIetY.Org
BrooksBeLl.COM
Puc-mPLs.oRG
WWW.precept.oRG
TANgEntIA.com
WWw.ELEctRICsandLigHtiNG.cOM
tRuebijOuX.Ca
pAthWaYs.ciO.Com
SEndhOpENow.oRg
WwW.IiSmE.orG
WWW.MOrneaUSHePeLl.Com
tEsT.hOgueSf.Com
BLoG.MOrNeauSHEPell.CoM
STUdeNTs.mBAcAreERS.WHArToN.UPeNn.edu
CEceXchaNgE.Com
WWw.TEsT.HogUESf.Com
eLEctriCSaNdligHTInG.com
GETOutDOOrScOLOrAdo.coM
godAY.GETOUTDoorSCoLoRadO.ORg
eLeCtricsaNDLIgHTInG.bIz
TesT.fIm.nOrTheAStErn.EdU
www.COpIOUsManAGEmENT.CoM
HaNdsOff.Msf.ORG
Www.WhOLEgRAInScOUNciL.Org
wwW.mMitiOwA.com
Www.ElectRICsANDLIGhTiNG.BIZ
GEToUtdOorSCOloRADo.oRG
www.tAngeNtia.com
wWw.MmsaLES.neT
aMeRICanpOssIbILItiEs.oRg
DonaTeYourvoIce.wOrLdvISion.orG.sG
WhOLegrAINScOuNCil.ORG
VTSMoKEaNDCuRe.COm
Www.gETouTDOORSCOLORADo.cOM
Test.UpTOWNalmaNAC.cOM
tEsT.MorNeAuShEPelL.cOm
wWW.TrUEbIJOUX.CA
WWw.puC-MN.Org
nGod.gEToUtDOORsCOLOrAdo.cOM
mmsAleS.net
sPonsor.sEndhopeNOW.org
moRNEAUsHEPelL.cOm
WwW.MMvaf.COM
wWw.lSaDC.oRG
WWW.whitEbirchsofTwarE.com
JoiN.IGnIteDucAtioN.org
CHECkPlEase.WTTW.COm
WWw.REsIstaNCedAShBOArd.CoM
MmvAF.coM
meGaTRAcks.LioneL.com
ClOUD.CLAtEST.cOm
BcC.INtErcityTRaNSit.coM
wWW.HEreToHELp.bc.ca
MMiTIowa.coM
PremiERTaTtoOiNK.CoM
triNItYveNTuReS.CoM
wWW.cLOud.CLaTESt.COM
WWW.PRAKTiJk.inFo
5725851488354304-fE3.PaNtHEONsiTe.iO
GoDAY.GeToutdoorScOLoRADo.COm
skILLFuL.com
preCePT.ORg
wwW.trIniTYvenTUrEs.COm
pUc-mn.oRG
wWw.preMiErTaTToOinK.COM
Iisme.org
FiM.noRThEaSTERn.edU
LINGuiStiCsoCietY.OrG
HeLP.CHI.ac.uk
WwW.bRoOkSbelL.com
PortAL.SubLUxATion.com
DeV-CHecKPleaSe.wtTW.CoM
ELeCTRIcsanDLIgHtiNG.CO.UK
imAginGtecH.neT
DeV.FIrEroaddIgItal.cOm
wWW.ceceXCHAngE.com
TeST-interaCTiVE.WTTW.com
LsAdc.oRg
PrODTEst.pANtHEoNdNstESTDOMAin.Com
CiviCrm.IIsme.oRG
TEST.grONau-iT-CLoUd-cOMpuTING.dE
Www.GEToUTDOoRScoloRAdO.oRg
PreviEW.CeceXChanGe.cOM
WwW.SeNdHopeNow.Org
www.ELectRIcSAndliGHtIng.CO.uk


WWW.xlMr.UK
VistaIMAgINgseRVICEs.cOM
TEst.PLaiNHEaLTh.cOm
Www.MoNtGOMeRyvIKINgsAlumnI.oRg
dEVEthiCaLTrADE.CrS.ORG
Www.wAVerLEyLANE.net
Www.ISoj.orG
XLmr.me.uk
WWw.XlmR.org
wWW.mCdOnaLdjoNEshomES.com.aU
xLmr.OrG.Uk
www.WAverleyLanE.infO
IsD191.orG
wWW.WaVerlYlANE.co.uk
XlMR.Net
mONtyLInk.Com
Www.GoLdENstaTeluMber.COM
xLMr.OrG
WwW.HouSeLENS.cOM
AjcloSANGeLES.org
NED30yeArs.com
NeD30undER30.ORG
DEv.AsTekWALLcoVErInG.cOm
USMicrOPRoDuCts.COM
wWw.pORTlAndsfiVe.iNFo
PanDRDeSiGnco.COM
WwW.waVeRleyLaNE.oRG
Www.GOABsinC.COm
Tcell.io
www.libertYPArKS.cOM
wwW.portlAND5theaTerS.OrG
WWw.usmIcrOprODuctS.Com
ethICAlTrADE.CrS.org
ExpLoREBalboaislanD.COm
WWw.monTyliNk.com
MCDonalDjoneshOmeS.COm.au
www.montgomERyhiGHeducaTioNfouNdaTIon.org
goLdEnsTatELumBeR.Net
GoLDeNStatELumber.orG
wWW.XLmr.ME.uK
wwW.HeehaW.COm
goLDensTATeLUMbeR.uS.coM
WWw2.vISTaIMaGinGseRVicEs.com
wWw.gOlDenStatEluMbER.oRG
wWw.WSMonLiNe.com
5757487680585728-fE4.paNtheonsitE.Io
NeD30UNder30.NeT
deV.wesTBuRGUndyWinEcollECtiVe.coM
www.gOldeNstaTelumbEr.uS.COm
mail.ViStAimaGiNgsERVICES.CoM
wWW.GoldENSTATElUmBEr.NET
Www.porTLandSFIVe.orG
Www.pORTLANdFIVE.org
wWW.Isd191.Org
goldEnStaTELuMber.uS
www.XLmR.CO.uk
BenneTtoFFIcEEQUiP.CoM
ftp.vISTaiMagiNGsErviCeS.coM
wWw.eXiGEr.com
WwW.PORtLanDsFiVe.NeT
wWW.POrtlANdFiVe.coM
dEV.MonTgoMEryviKingSAlumnI.ORG
www.PGHdENtalsPA.coM
GOABSinc.com
WWW.POrtLanDFivE.InFo
NocCIOladESIgN.coM
wWW.pOrTLaND5theaTErs.iNFo
hOuSelENs.Com
AcLUSD.OrG
saNDBOX.tEcHENhancEDLIFE.com
MONTgOmeRyHigheduCaTIOnFOUNDAtiOn.Org
WWw.golDENSTAtelumBer.bIZ
EXIGeR.cOM
Www.aCLUSD.oRg
ISoJ.Org
MonTGomeRYvIKiNgsalUMNI.oRG
AClUGa.ORG
wWw.ViStaImaGinGSErVICES.cOm
WWW.POrTLANDFiVe.neT
XlMr.Uk
wWW.ExPlOREbALbOaIsLAND.cOm
WwW.XlmR.orG.uK
Ned30UnDeR30.Com
WwW.AClUgA.Org
WWw.XlMr.neT
TEst.tippVet.Com
WwW.pOrtlAND5thEaTERS.NET
GOlDensTATEluMbER.InfO
wwW3.viSTaiMAGinGSErVIceS.cOm
WWW.noccIoladESigN.COm
XLmr.co.UK
ProjeCTGrAd.MontYLinK.cOM
wsMonlinE.COM
wWW.AJcLOsaNgElES.ORg
LiBErtYpaRKs.COM
www.gOLDeNsTaTELuMber.INfO
Www.gOLDENstATeLUmBEr.uS
HeEhaW.Com
wWw.BENnettofFiceEqUIp.Com
Www.PORtlANdsFIVe.Com


We’ve tried auditing the certificates to make sure they comply with LE requirements (CAA/DNAME). I’m not sure why we’re getting 500s.


#9

Adding a 9th cert to the list:

PapoWErSWitcH.org
WwW.sUIciDe6.CoM
www.FrienDSWHoCounT.CoM
WwW.quIEtRoCk.COM
sCLt.in
wWW.CbCsTEELBuILdinGS.CoM
WWW.AClUmONTana.OrG
WwW.FAmILYVacaTION-Vt.COM
www.wOODstOCKinn.cOm
en.DEv.iNVESTINboGoTA.org
wwW.aCluSC.oRg
gOEE.asU.Edu
ExTRAneT.PapOWersWItch.cOm
aclUsC.ORG
AsUEVeNtS.ASU.edU
WWW.tHeDAtacenTErgUrU.uS
wWW.reDrOOsteRvt.Com
ANTidopInGlEaRnINGHUb.oRG
www.meEtINNewEnGlAND.neT
wWW.meeTiNVT.com
ajCloSaNGELEs.oRg
wWw.vt-VaCatIoN.Com
HABItATEbSV.orG
yosEMIteiNstItUte.naturEBRIdgE.org
StAgInG.THEMxgrOuP.com
sUMmErSciencE.NaTurebRidge.oRg
aSUnOw.asu.Edu
Www.aBaXis.COm
INveStInbOGOtA.oRG
www.sUIcIdesix.coM
ExpLOreBaLBoAiSLand.COm
WwW.thErEDrOostERInWoodStoCk.COm
www.FOx21OnLInE.COm
AbaXIs.COm
wWW.THedATacenTERGURu.nET
tHUndErBIRd.ASU.EdU
pA-POWEr-sWiTCH.OrG
D9BYiEXufI9YW.cLoUDfront.neT
Www.mEetiNthEgreEnmouNtAiNs.COM
tEST.AmeriCanTasKFOrcE.OrG
TEEM.nAtUReBridGE.ORg
DEv.INVestINbOgoTA.ORG
CbcsTEElBUILdinGs.CoM
WWw.VAcATioN-VT.coM
WWW.paNDrdeSigNco.coM
WWw.AMeRIcaNtASkforCE.oRg
CoMMENTdevEnIrADULTe.cOM
eNgliSh.InVEsTINBOGOTA.OrG
MArTins.LAB.asu.eDu
TESt.ATfp.NEt
WWW.atFp.NET
LiVe.Atfp.NEt
cRmhELpdeSK.NaturEBRiDGe.oRg
OlymPIcPARkINSTituTe.NatUreBrIDge.oRg
eNtErmeDIA.COm
Dev.ATFp.neT
QuietroCK.coM
frIEnDsWHOcOUnt.cOm
En.inVESTInBogota.orG
www.ENterMeDIa.COm
5186145897938944-Fe4.PAnTHEoNSite.IO
aCLUmontana.orG
naTuREBRIdgE.Org
shoRtLiNKS.NAtuReBriDge.orG
wOODstOcKINNSpA.Com
www.nAtUREBRiDGe.OrG
yNI.naTuREBrIdgE.OrG
aMeRICaNtaSkFOrcE.OrG
PaPOWerSWItcH.CoM
D6iWo6kNwpGWa.CLoUdFrONT.neT
WWw.friENdsWhOCoUnt.Org
WWw.aNTidOpiNGleArnINGhUb.orG
Fox21oNLinE.com
pA-power-SWiTch.CO
WWW.WoodsTOckInnSPa.coM
Www.hAbitAtEbSv.ORg
hICONFerENCECEnter.naTurEbRIdGe.ORg
BlocKCHain.lab.ASu.edu
eS.dEv.InVesTINBogOtA.ORG
wWW.VeRdIct.Co.UK
ES.invESTInboGotA.Org
v1aRCHive.pgPf.orG
Www2.nATURebRidgE.Org
DeV.ameRicANtAsKFoRce.org
swrOBOtIcs.EngINEeRING.ASU.eDU
wWw.THedatACeNTErGuRU.ORg
pA-pOWeR-sWiTCH.CoM
WEB.aSU.Edu
PaPoWERswITcH.neT
WwW.tHedataCeNTeRguru.Info
wWW.vTFaMIlYvaCATIOn.Com
LifeiNsURANceInteRNATioNal.cOm
aTfp.Net
eS.tesT.iNvEstINBOgoTA.OrG
LIVE.aMErIcanTasKforCE.OrG
eN.tEST.iNVESTiNbogOTa.OrG
m.PapoweRsWiTCH.COM
WWW.SCLt.in
HEaDLandsiNstItUtE.nAtuREBRIDge.org
PA-PoWER-SwITch.NEt


#10

Hi @jsha,

Is it known what the latency issue is? I was guessing it was the rate limit check. If it is that, is the database not able to handle the load or is it that doing 100 rate limit checks serially exceeds a timeout? I’d be happy to write the parallelization of the lookups if that would help.

Cheers,
Ben

P.S., Where can I find what features LE is currently running? I’m curious if features.AllowRenewalFirstRL has been enabled.


#11

You have good instincts! We’re pretty sure that’s where the error lies. Most of our queries can be clumped up so the round trips don’t scale with the number of names on a certificate. However, the certificates per name rate limit queries do scale that way. Our current plan is to parallelize those queries. I’m working on that this week. Thanks for your offer of help! If we get hung up on other urgent work I may take you up on it, but for now assume we’ll do it.

Generally speaking, the feature flags in test/config are what’s in prod, while test/config-next is stuff that may be in prod soonish. Unfortunately AllowRenewalFirstRL is disabled because of some performance problems.


#12

Is there any update on this issue? Is it likely to be fixed in the near future?


#13

We’ve landed a code change to increase parallelism for the necessary RPCs, and will be enabling it via config on Thursday of this week.


#14

Did the change to increase parallelism get enabled?


#15

It hasn’t yet, sorry about that. Looks like a ticket got dropped along the way; I’ll check and see what happened.


#16

Sorry to keep asking, but do you have an update on this? We are at the critical part of our development and need to know whether this is going to be possible or we need to find an alternative solution.


#17

We roll out new Boulder deploys and config changes most Thursdays. We skipped last Thursday, so the next window for this change is this coming Thursday. I’ll double check that the change will make it in.


#18

Is it possible to get an update on this?


#19

Hi @frapps,

The config change went to production yesterday - are you still seeing elevated 500s from the new-cert endpoint for 100 SAN CSRs?


#20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.