Where is apache plugin getting this domain from?

On a clean install of certbot with Ubuntu 20.04 I run certbot --apache I am getting output like this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: example.com
2: mail123.example.org
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):

Oddly, the second domain option doesn't appear anywhere is any apache config file that I can find. I tried a few commands to search the config files like this:

$ grep -r mail123 /etc
$ grep -r mail123 /lib
$ grep -r mail123 /var    
$ grep -r mail123 /usr
$ grep -r mail123 /home

And came up with nothing.

I see in the logs a reference to:

File "/usr/lib/python3/dist-packages/certbot/main.py", line 1127, in run
domains, certname = _find_domains_or_certname(config, installer)

But I don't know python well enough to understand the logic involved. How can I find out where the apache plugin is getting this domain from?

2 Likes

The subdomain found by certbot states mail1, but you're "grep"-ping for mail123?

In any case, it gets its hostnames from the Apache configuration, no where else.

2 Likes

Just a typo in post question.. I've updated it.

2 Likes

Running apachectl -S might reveale the virtualhost in question.

3 Likes

It does indeed show up there as:

*:80                   mail123.example.org (/etc/apache2/sites-enabled/000-default.conf:1)

But 000-default.conf is only exactly what you'd expect and doesn't contain any reference to said domain.

1 Like

Apache doesn't just come up with that hostname. It has to come from somewhere in that configuration file.

Could you paste it here? Please put three backticks (```) on a separate line above and below the contents of the file for better readability.

4 Likes

It can, if there is no ServerName and the VirtualHost contains a bind address that Apache can turn back into a domain name:

If no ServerName is specified, the server attempts to deduce the client visible hostname by first asking the operating system for the system hostname, and if that fails, performing a reverse lookup on an IP address present on the system.

Pretty crappy behavior; just use ServerName.

edit: better citation

6 Likes

As an extreme investigative tool you could try

sudo strace -s 65536 -e open,read apachectl -S 2>&1 >/dev/null | less

and see exactly what file descriptor that string came in on (searching /mail123 in less). Assuming that apachectl runs a subprocess, you might also need to add a -f to the strace command line to trace the subprocess as well.

If that still doesn't find it, you could leave out the -e open,read part and then you're looking at every operating system call that the process makes. Hopefully that will be able to find it.

6 Likes

I guess "coming up with" can be interpreted in that way indeed.. I meant Apache doesn't just randomly make up hostnames. There has to be something behind it. I should have used better wording.

4 Likes

I understand why you'd ask but with all due respect, I'm not pasting the default vhost file here, it's a waste of time and If you've ever installed apache you already know what's in there anyway.

This last one was the only thing the yielded something - which was along these lines:

[pid 188148] recvfrom(7, "\330X\201\200\0\1\0\1\0\0\0\1\003XXX\003XXX\003XXX\003XXX\7in-addr\4arpa\0\0\f\0\1\300\f\0\f\0\1\0\0\3N\0\26\5mail123\nexample\3org\0\0\0)\377\326\0\0\0\0\0\0", 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28->16]) = 91

Looks an awful lot like server is pulling that domain from the LAN DNS or something like that.

1 Like

In that case you're on your own figuring out why that Apache configuration file presents you that hostname.

3 Likes

Seems pretty consistent with the earlier snippet I pasted from ServerName documentation.

That recv syscall doesn't contain a full valid DNS message, but from what we can see, it appears to be a reverse DNS lookup for 216.x.x.233, which is exactly what is described:

... and if that fails, performing a reverse lookup on an IP address present on the system.

and that address does indeed have a PTR record for mail1.<redacted>.org.

3 Likes

Seems related for sure but for the following things:

  1. Disabling the default virtual host yields same result
  2. Adding ServerName directive to virtual host yields same result
  3. My server has a system hostname clearly defined
  4. The LAN IP 127.0.0.53 appears nowhere in /etc/hosts or any config file I'm aware of

Also consider after doing the first two things the unknown domain does in fact disappear out of apachectl -S output, but still continues to show up in certbot's list of domains. It's not really a problem so much as a mystery at this point.

2 Likes

Bingo - so the WAN IP has PTR record and that's where this comes from. I'm still a bit mystified by why certbot isn't seeing our hostname and why there's a PTR lookup at all, but thanks, you've pretty much answered this!

3 Likes

The nature of debugging is that there is that usually there's something true that you believe to be false, and a good debugging process involves particularly verifying everything you assume must be normal and expected. As some concrete examples of why pasting the vhost file would be informative:

  • Different distributions ship different default config files.
  • Different versions of distributions ship different default config files.
  • Some users use tools built on top of the distro that generate different default config files.
  • Some users make modifications, and then forget they modified them.

I'm glad folks were able to help figure this out with seeing the config file, but I want to emphasize that it was reasonable and correct of @Osiris to ask for it. When debugging, doubt everything (in a targetted way). :slight_smile:

4 Likes

Or the opposite, which somehow illustrates the same point and forms a maxim I've pounded into the heads of many a researcher...

What you don't know can hurt you, but what you know that is wrong will kill you.

1 Like

Haha, ok point well taken. Here it is!

<VirtualHost *:80>
	# The ServerName directive sets the request scheme, hostname and port that
	# the server uses to identify itself. This is used when creating
	# redirection URLs. In the context of virtual hosts, the ServerName
	# specifies what hostname must appear in the request's Host: header to
	# match this virtual host. For the default virtual host (this file) this
	# value is not decisive as it is used as a last resort host regardless.
	# However, you must set it for any further virtual host explicitly.
	#ServerName www.example.com

	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/html

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
3 Likes

Sorry if I came off rude. I appreciate the effort you made to look at this. Thanks

5 Likes

I personally can see both sides, but good intentions all around. We all know which road those pave... :smiling_imp:

Overall though, I don't think there are any hard feelings. I've personally hammered Osiris worse than that and he hasn't cracked yet. :hammer:

Well sometimes he cracks. :thinking: Wisely. :grin:

1 Like

image
Where do I pay to buy more likes?!?!?!? - LOL

2 Likes