Hello
We have a mattermost server running as a set of docker containers (https://github.com/mattermost/mattermost-docker) on ubuntu(16) server machine with ip: 145.239.86.144 and with mattermost.iterative.pl domain.
After a year and half of running this setup without any problem today it suddenly failed to renew cert via acme with following msg:
Error initializing issuer: 403 urn:ietf:params:acme:error:rateLimited: Your IP, 145.239.86.144, has been blocked due to ridiculously excessive traffic. Once this is corrected you may request this be reviewed on our forum https://community.letsencrypt.org
After a bit more digging we found out that in directory with certs there are plenty of generated for domains which clearly were not ours, ie:
pkakxmtgtjochtm.vixera.eu+rsa
vixera.eu
vixera.eu+rsa
At the moment we are not 100% sure how that happen, but it seems that version of mm we had been running got some vulnerability, which allowed attacker to manipule mattermost to generate those certs (and as a result flood lets encrypt server with requests). Note that we hadn’t been using reverse proxy, but just mattermost server directly and its internal acme lets encrypt setup along with it.
As a remedy, for now we had:
- removed all unwanted cert files
- updated mattermost app to latest version
- changed config to use nginx as a reverse-proxy along with SSL termination at its stage.
Due to block we had to manually generate new cert, but obviously ideally we would pref to enable acme on nginx or setup certbot auto-renew.
To our best knowledge remedies we did should stop any excessive traffic from happening further. So could you please unblock our IP address 145.239.86.144?
cheers,
Krzysztof