.well-known/acme-challenge attempts on validated domain

e-comBR · December 14, 2017, 11:28am

My server keeps being hit by .well-known/acme-challenge attempts on a particular domain (sistema.fetrafparana.org.br). However, this domain is already validated (and recently the cert was renewed, and working 100%) by tls.sni.

There are other domains in this server, all of them using Let’s Encrypt certs, and none of them had the .well-know attempts.

What can I do to fix this issue?

My domain is: sistema.fetrafparana.org.br

I ran this command: (none)

It produced this output: (n/a)

My web server is (include version): nginx/1.10.2

The operating system my web server runs on is (include version): CentOS 6.8

My hosting provider, if applicable, is: (n/a)

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): (n/a)

Osiris · December 14, 2017, 11:58am

Perhaps somebody is trying to get a certificate for your domain, without “consent”.

Luckily, without a valid token, that can’t happen

I don’t think you can do anything against it. I guess someone thinks your domain is important enough to try somethink like this

e-comBR · December 14, 2017, 12:13pm

Odd, that shouldn’t be the case. I expected some misconfiguration on my part.

Is there a way to get the IP address of whoever started the request?

e-comBR · December 14, 2017, 12:15pm

More details: we get the request every hour, surely it’s some automated tool.

Osiris · December 14, 2017, 12:16pm

I don’t think so…

But there are a few things you can check of course. For example, is it always the same token which is requested? Or is it different per request?

Also from which IP address is the (direct) request coming from? Perhaps you’ve entered the URL in this community and was it crawled by a search engine crawler or something.

And what is the user agent of the requests?

e-comBR · December 14, 2017, 12:33pm

Sorry Osiris, I should have pasted such details earlier:

It’s always a different token. IP seems to be from Let’s Encrypt:

sistema.fetrafparana.org.br.access.log:64.78.149.164 - - [14/Dec/2017:04:58:07 -0200] "GET /.well-known/acme-challenge/LvAY4zS2RiHXMwjif3MHnv0FNp1e7lde-fKUR7fqVZI HTTP/1.1" 301 185 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.149 -
sistema.fetrafparana.org.br.access.log:64.78.149.164 - - [14/Dec/2017:04:58:17 -0200] "GET /.well-known/acme-challenge/LvAY4zS2RiHXMwjif3MHnv0FNp1e7lde-fKUR7fqVZI HTTP/1.1" 499 0 "http://sistema.fetrafparana.org.br/.well-known/acme-challenge/LvAY4zS2RiHXMwjif3MHnv0FNp1e7lde-fKUR7fqVZI" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 9.230 -
sistema.fetrafparana.org.br.access.log:64.78.149.164 - - [14/Dec/2017:05:58:11 -0200] "GET /.well-known/acme-challenge/9KMZgDmMhpAH09bC8XeGZhYoMjUE_FHuJ1btZzgAWj0 HTTP/1.1" 301 185 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.149 -
sistema.fetrafparana.org.br.access.log:64.78.149.164 - - [14/Dec/2017:05:58:20 -0200] "GET /.well-known/acme-challenge/9KMZgDmMhpAH09bC8XeGZhYoMjUE_FHuJ1btZzgAWj0 HTTP/1.1" 499 0 "http://sistema.fetrafparana.org.br/.well-known/acme-challenge/9KMZgDmMhpAH09bC8XeGZhYoMjUE_FHuJ1btZzgAWj0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 9.237 -
sistema.fetrafparana.org.br.access.log:64.78.149.164 - - [14/Dec/2017:06:58:09 -0200] "GET /.well-known/acme-challenge/IY3mML5TawhnzIB4QT7wihM9LIYPEGZY83bmLvrCNSk HTTP/1.1" 301 185 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.149 -
sistema.fetrafparana.org.br.access.log:64.78.149.164 - - [14/Dec/2017:06:58:18 -0200] "GET /.well-known/acme-challenge/IY3mML5TawhnzIB4QT7wihM9LIYPEGZY83bmLvrCNSk HTTP/1.1" 499 0 "http://sistema.fetrafparana.org.br/.well-known/acme-challenge/IY3mML5TawhnzIB4QT7wihM9LIYPEGZY83bmLvrCNSk" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 9.233 -

Osiris · December 14, 2017, 12:38pm

Do you have a cronjob or systemd timer running which might related to a ACME client?

e-comBR · December 14, 2017, 12:51pm

There’s auto renewal set up in crontab:

# crontab -e
46 */12 * * * /usr/local/sbin/renovar_certificados_letsencrypt.sh

That .sh script set’s up the path variable and invokes /usr/local/sbin/certbot-auto renew --quiet, nothing else.

Looking at /var/log/crontab around the time of the requests, there’s nothing suspicious.

This system does not run systemd. I’m not aware of other timers, nor how to check them.

# grep well /var/log/letsencrypt/letsencrypt.log*
returns nothing

Osiris · December 14, 2017, 12:53pm

What’s the complete contents of letsencrypt.log?

Although your cronjob isn’t set up for hourly renewal… So it’s unlikely this script is causing the problems.

bytecamp · December 14, 2017, 12:56pm

This relates to a user crontab.
What about the system's crontab (if any) /etc/crontab or files below /etc/cron.d ?

e-comBR · December 14, 2017, 1:56pm

If certbot doesn’t schedule tasks by itself, then that’s the only one I configured. Anyway, here’s more:

cat /etc/crontab shows:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

ls /etc/cron.hourly

0anacron awstats inn-cron-nntpsend inn-cron-rnews

cat /etc/cron.hourly/*

#!/bin/bash
# Skip excecution unless the date has changed from the previous run 
if test -r /var/spool/anacron/cron.daily; then
    day=`cat /var/spool/anacron/cron.daily`
fi
if [ `date +%Y%m%d` = "$day" ]; then
    exit 0;
fi

# Skip excecution unless AC powered
if test -x /usr/bin/on_ac_power; then
    /usr/bin/on_ac_power &> /dev/null
    if test $? -eq 1; then
    exit 0
    fi
fi
/usr/sbin/anacron -s
#!/bin/bash
exec /usr/share/awstats/tools/awstats_updateall.pl now         -configdir="/etc/awstats"         -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" >/dev/null
exit 0
#!/bin/sh
/sbin/chkconfig innd || exit 0
runuser - news -c "unset LANG; unset LC_COLLATE; /usr/lib/news/bin/nntpsend"
#!/bin/sh
/sbin/chkconfig innd || exit 0
runuser - news -c 'unset LANG; unset LC_COLLATE; /usr/lib/news/bin/rnews -U'

e-comBR · December 14, 2017, 2:00pm

cat /var/log/letsencrypt/letsencrypt.log

2017-12-14 02:46:15,065:DEBUG:certbot.main:certbot version: 0.20.0
2017-12-14 02:46:15,065:DEBUG:certbot.main:Arguments: ['--quiet']
2017-12-14 02:46:15,065:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2017-12-14 02:46:15,182:DEBUG:certbot.log:Root logging level set at 30
2017-12-14 02:46:15,182:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-12-14 02:46:15,333:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x9076fcc> and installer <certbot.cli._Default object at 0x9076fcc>
2017-12-14 02:46:15,334:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x9072e8c>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x9072e0c>, apache=<certbot.cli._Default object at 0x907902c>, apache_challenge_location=<certbot.cli._Default object at 0x90798ac>, apache_ctl=<certbot.cli._Default object at 0x90799cc>, apache_dismod=<certbot.cli._Default object at 0x90796cc>, apache_enmod=<certbot.cli._Default object at 0x907968c>, apache_handle_modules=<certbot.cli._Default object at 0x907990c>, apache_handle_sites=<certbot.cli._Default object at 0x907996c>, apache_init_script=<certbot.cli._Default object at 0x9079a2c>, apache_le_vhost_ext=<certbot.cli._Default object at 0x907972c>, apache_logs_root=<certbot.cli._Default object at 0x907984c>, apache_server_root=<certbot.cli._Default object at 0x907978c>, apache_vhost_root=<certbot.cli._Default object at 0x90797ec>, authenticator=<certbot.cli._Default object at 0x9076fcc>, break_my_certs=<certbot.cli._Default object at 0x907626c>, cert_path=<certbot.cli._Default object at 0x9076b6c>, certname=<certbot.cli._Default object at 0x9072acc>, chain_path=<certbot.cli._Default object at 0x907690c>, checkpoints=<certbot.cli._Default object at 0x9076d0c>, config_dir=<certbot.cli._Default object at 0x9076c6c>, config_file=None, configurator=<certbot.cli._Default object at 0x9076fcc>, csr=<certbot.cli._Default object at 0x9076e2c>, debug=<certbot.cli._Default object at 0x907608c>, debug_challenges=<certbot.cli._Default object at 0x90760ec>, deploy_hook=<certbot.cli._Default object at 0x907668c>, dialog=None, directory_hooks=<certbot.cli._Default object at 0x907670c>, dns_cloudflare=<certbot.cli._Default object at 0x90792ac>, dns_cloudxns=<certbot.cli._Default object at 0x907930c>, dns_digitalocean=<certbot.cli._Default object at 0x907936c>, dns_dnsimple=<certbot.cli._Default object at 0x90793cc>, dns_dnsmadeeasy=<certbot.cli._Default object at 0x907942c>, dns_google=<certbot.cli._Default object at 0x907948c>, dns_luadns=<certbot.cli._Default object at 0x90794ec>, dns_nsone=<certbot.cli._Default object at 0x907954c>, dns_rfc2136=<certbot.cli._Default object at 0x90795ac>, dns_route53=<certbot.cli._Default object at 0x907960c>, domains=<certbot.cli._Default object at 0x9072a8c>, dry_run=<certbot.cli._Default object at 0x9072b0c>, duplicate=<certbot.cli._Default object at 0x9072eec>, eff_email=<certbot.cli._Default object at 0x9072c2c>, email=<certbot.cli._Default object at 0x9072bcc>, expand=<certbot.cli._Default object at 0x9072cec>, force_interactive=<certbot.cli._Default object at 0x9072a4c>, fullchain_path=<certbot.cli._Default object at 0x9076f4c>, func=<function renew at 0x8d7f79c>, hsts=<certbot.cli._Default object at 0x90763ac>, http01_address=<certbot.cli._Default object at 0x907622c>, http01_port=<certbot.cli._Default object at 0x90761ec>, ifaces=<certbot.cli._Default object at 0x9076f8c>, init=<certbot.cli._Default object at 0x9076fec>, installer=<certbot.cli._Default object at 0x9076fcc>, key_path=<certbot.cli._Default object at 0x907686c>, logs_dir=<certbot.cli._Default object at 0x9076cec>, manual=<certbot.cli._Default object at 0x90791ac>, manual_auth_hook=<certbot.cli._Default object at 0x907966c>, manual_cleanup_hook=<certbot.cli._Default object at 0x9079aec>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x9079b4c>, max_log_backups=<certbot.cli._Default object at 0x90729cc>, must_staple=<certbot.cli._Default object at 0x90762ec>, nginx=<certbot.cli._Default object at 0x90790cc>, nginx_ctl=<certbot.cli._Default object at 0x9079c0c>, nginx_server_root=<certbot.cli._Default object at 0x9079a8c>, no_bootstrap=<certbot.cli._Default object at 0x9072fac>, no_self_upgrade=<certbot.cli._Default object at 0x9072f6c>, no_verify_ssl=<certbot.cli._Default object at 0x907612c>, noninteractive_mode=<certbot.cli._Default object at 0x9072a0c>, num=<certbot.cli._Default object at 0x90767ac>, os_packages_only=<certbot.cli._Default object at 0x9072f2c>, post_hook=<certbot.cli._Default object at 0x907660c>, pre_hook=<certbot.cli._Default object at 0x90765ac>, pref_challs=<certbot.cli._Default object at 0x907656c>, prepare=<certbot.cli._Default object at 0x9076d6c>, quiet=True, reason=<certbot.cli._Default object at 0x9076cac>, redirect=<certbot.cli._Default object at 0x907632c>, register_unsafely_without_email=<certbot.cli._Default object at 0x9072b4c>, reinstall=<certbot.cli._Default object at 0x9072cac>, renew_by_default=<certbot.cli._Default object at 0x9072d8c>, renew_hook=<certbot.cli._Default object at 0x907664c>, renew_with_new_domains=<certbot.cli._Default object at 0x9072dcc>, rsa_key_size=<certbot.cli._Default object at 0x90762ac>, server=<certbot.cli._Default object at 0x90768ec>, staging=<certbot.cli._Default object at 0x907604c>, standalone=<certbot.cli._Default object at 0x907914c>, standalone_supported_challenges=<certbot.cli._Default object at 0x9079c6c>, staple=<certbot.cli._Default object at 0x90764ac>, strict_permissions=<certbot.cli._Default object at 0x907652c>, text_mode=<certbot.cli._Default object at 0x907284c>, tls_sni_01_address=<certbot.cli._Default object at 0x90761ac>, tls_sni_01_port=<certbot.cli._Default object at 0x907616c>, tos=<certbot.cli._Default object at 0x9072e4c>, uir=<certbot.cli._Default object at 0x907642c>, update_registration=<certbot.cli._Default object at 0x9072b8c>, user_agent=<certbot.cli._Default object at 0x907680c>, user_agent_comment=<certbot.cli._Default object at 0x9076eac>, validate_hooks=<certbot.cli._Default object at 0x90766cc>, verb='renew', verbose_count=<certbot.cli._Default object at 0x907250c>, webroot=<certbot.cli._Default object at 0x907922c>, webroot_map=<certbot.cli._Default object at 0x9079d4c>, webroot_path=<certbot.cli._Default object at 0x9079bac>, work_dir=<certbot.cli._Default object at 0x9076aac>)
2017-12-14 02:46:15,391:INFO:certbot.renewal:Cert not yet due for renewal
2017-12-14 02:46:15,420:INFO:certbot.renewal:Cert not yet due for renewal
2017-12-14 02:46:15,475:INFO:certbot.renewal:Cert not yet due for renewal
2017-12-14 02:46:15,520:INFO:certbot.renewal:Cert not yet due for renewal
2017-12-14 02:46:15,532:INFO:certbot.renewal:Cert not yet due for renewal
2017-12-14 02:46:15,532:DEBUG:certbot.renewal:no renewal failures

Yesterday log is similar, except there are two runs, as scheduled.

mnordhoff · December 14, 2017, 8:57pm

It seems more likely that, for example, a misconfigured ACME client was left on one of your test servers, than that someone else is maliciously (and unsuccessfully) trying to get a certificate for your site… Still, it’s possible.

The Let’s Encrypt staff can look up the IP address(es) and account(s) doing this, but I’m unsure what they’d be willing to share, for privacy reasons.

system · January 13, 2018, 8:57pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.