Oddly, the second domain option doesn't appear anywhere is any apache config file that I can find. I tried a few commands to search the config files like this:
It can, if there is no ServerName and the VirtualHost contains a bind address that Apache can turn back into a domain name:
If no ServerName is specified, the server attempts to deduce the client visible hostname by first asking the operating system for the system hostname, and if that fails, performing a reverse lookup on an IP address present on the system.
and see exactly what file descriptor that string came in on (searching /mail123 in less). Assuming that apachectl runs a subprocess, you might also need to add a -f to the strace command line to trace the subprocess as well.
If that still doesn't find it, you could leave out the -e open,read part and then you're looking at every operating system call that the process makes. Hopefully that will be able to find it.
I guess "coming up with" can be interpreted in that way indeed.. I meant Apache doesn't just randomly make up hostnames. There has to be something behind it. I should have used better wording.
I understand why you'd ask but with all due respect, I'm not pasting the default vhost file here, it's a waste of time and If you've ever installed apache you already know what's in there anyway.
This last one was the only thing the yielded something - which was along these lines:
Seems pretty consistent with the earlier snippet I pasted from ServerName documentation.
That recv syscall doesn't contain a full valid DNS message, but from what we can see, it appears to be a reverse DNS lookup for 216.x.x.233, which is exactly what is described:
... and if that fails, performing a reverse lookup on an IP address present on the system.
and that address does indeed have a PTR record for mail1.<redacted>.org.
Seems related for sure but for the following things:
Disabling the default virtual host yields same result
Adding ServerName directive to virtual host yields same result
My server has a system hostname clearly defined
The LAN IP 127.0.0.53 appears nowhere in /etc/hosts or any config file I'm aware of
Also consider after doing the first two things the unknown domain does in fact disappear out of apachectl -S output, but still continues to show up in certbot's list of domains. It's not really a problem so much as a mystery at this point.
Bingo - so the WAN IP has PTR record and that's where this comes from. I'm still a bit mystified by why certbot isn't seeing our hostname and why there's a PTR lookup at all, but thanks, you've pretty much answered this!
The nature of debugging is that there is that usually there's something true that you believe to be false, and a good debugging process involves particularly verifying everything you assume must be normal and expected. As some concrete examples of why pasting the vhost file would be informative:
Different distributions ship different default config files.
Different versions of distributions ship different default config files.
Some users use tools built on top of the distro that generate different default config files.
Some users make modifications, and then forget they modified them.
I'm glad folks were able to help figure this out with seeing the config file, but I want to emphasize that it was reasonable and correct of @Osiris to ask for it. When debugging, doubt everything (in a targetted way).
<VirtualHost *:80>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request's Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
#ServerName www.example.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
Wisely. 
