A Further Possible Bug With Fedora 30

Help
1

My domain is corequery.uk and I am trying to install SANS certificates on Fedora 30 with Apache 2.4 and full root access (command line). The whole process just hangs with the following log file output:

2019-07-04 21:18:25,129:DEBUG:certbot.main:certbot version: 0.35.1
2019-07-04 21:18:25,129:DEBUG:certbot.main:Arguments: ['--domains', 'exstocktra.de,insurgent.info,corequery.uk', '--apache']
2019-07-04 21:18:25,129:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-07-04 21:18:25,143:DEBUG:certbot.log:Root logging level set at 20
2019-07-04 21:18:25,143:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-07-04 21:18:25,144:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2019-07-04 21:18:37,458:DEBUG:certbot_apache.configurator:Apache version is 2.4.39
2019-07-04 21:19:09,817:ERROR:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.35.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.7/site-packages/certbot/main.py", line 1379, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.7/site-packages/certbot/main.py", line 1112, in run
    installer, authenticator = plug_sel.choose_configurator_plugins(config, plugins, "run")
  File "/usr/lib/python3.7/site-packages/certbot/plugins/selection.py", line 223, in choose_configurator_plugins
    authenticator = installer = pick_configurator(config, req_inst, plugins)
  File "/usr/lib/python3.7/site-packages/certbot/plugins/selection.py", line 24, in pick_configurator
    (interfaces.IAuthenticator, interfaces.IInstaller))
  File "/usr/lib/python3.7/site-packages/certbot/plugins/selection.py", line 105, in pick_plugin
    verified.prepare()
  File "/usr/lib/python3.7/site-packages/certbot/plugins/disco.py", line 253, in prepare
    return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)]
  File "/usr/lib/python3.7/site-packages/certbot/plugins/disco.py", line 253, in <listcomp>
    return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)]
  File "/usr/lib/python3.7/site-packages/certbot/plugins/disco.py", line 131, in prepare
    self._initialized.prepare()
  File "/usr/lib/python3.7/site-packages/certbot_apache/configurator.py", line 263, in prepare
    self.parser = self.get_parser()
  File "/usr/lib/python3.7/site-packages/certbot_apache/override_fedora.py", line 55, in get_parser
    self.version, configurator=self)
  File "/usr/lib/python3.7/site-packages/certbot_apache/override_fedora.py", line 86, in __init__
    super(FedoraParser, self).__init__(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/certbot_apache/parser.py", line 56, in __init__
    self.update_runtime_variables()
  File "/usr/lib/python3.7/site-packages/certbot_apache/override_fedora.py", line 91, in update_runtime_variables
    super(FedoraParser, self).update_runtime_variables()
  File "/usr/lib/python3.7/site-packages/certbot_apache/parser.py", line 145, in update_runtime_variables
    self.update_modules()
  File "/usr/lib/python3.7/site-packages/certbot_apache/parser.py", line 191, in update_modules
    matches = self.parse_from_subprocess(mod_cmd, r"(.*)_module")
  File "/usr/lib/python3.7/site-packages/certbot_apache/parser.py", line 205, in parse_from_subprocess
    stdout = self._get_runtime_cfg(command)
  File "/usr/lib/python3.7/site-packages/certbot_apache/parser.py", line 221, in _get_runtime_cfg
    stdout, stderr = proc.communicate()
  File "/usr/lib64/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib64/python3.7/subprocess.py", line 1681, in _communicate
    ready = selector.select(timeout)
  File "/usr/lib64/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
2

Does this hang if you run it manually?

httpd -t -D DUMP_MODULES
1 Like
3

@joohoi in this case the Apache subprocess spawned in get_runtime_cfg seems to be hanging and never finishing. I’m not sure why that should be so.

@Ah8fX6WA, could you try running this command and see what result you get?

sudo httpd -t -D DUMP_MODULES

4

Hi @Ah8fX6WA

your DNSSEC configuration is buggy ( https://check-your-website.server-daten.de/?q=corequery.uk ):

corequery.uk
	1 DS RR in the parent zone found

	1 RRSIG RR to validate DS RR found

	Algorithm: 8, 2 Labels, original TTL: 3600 sec, Signature-expiration: 18.07.2019, 15:29:21, Signature-Inception: 04.07.2019, 15:22:01, KeyTag 43056, Signer-Name: uk

	• Status: Good - Algorithmus 8 and DNSKEY with KeyTag 43056 used to validate the DS RRSet in the parent zone

	0 DNSKEY RR found

	Fatal error: Parent zone has a signed DS RR (Algorithm 13, KeyTag 3863, DigestType 2, Digest 
jW44Sq3f2VhZusXmkhB8cto9TpkRGu6vbGLiHwY1+5g=), but the destination DNSKEY 
doesn't exist or doesn't validate the DNSKEY RR set. No chain of trust created.

The uk zone has a DS, but your zone doesn't answer. Rechecked with DNSSEC Analyzer - corequery.uk to see, if my tool is buggy, there is the same:

uk
uk461×568 19.2 KB

Checked manual, your first nameserver (188.240.187.18) doesn't answer.

That's bad if a tool checks your authoritative ns.

5

Thanks for all the help with this. - I do not know why the DNS has gone bad because the server was only down for an hour or so and all is good now (no errors reported in the log files).

As for:

httpd -t -D DUMP_MODULES

…I tried it and (eventually) got the following:

Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 allowmethods_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_core_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_file_module (shared)
 authn_socache_module (shared)
 authz_core_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_owner_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 brotli_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 data_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dir_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 env_module (shared)
 expires_module (shared)
 ext_filter_module (shared)
 filter_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 log_config_module (shared)
 logio_module (shared)
 macro_module (shared)
 mime_magic_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 remoteip_module (shared)
 reqtimeout_module (shared)
 request_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 socache_redis_module (shared)
 socache_shmcb_module (shared)
 status_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 unique_id_module (shared)
 unixd_module (shared)
 userdir_module (shared)
 version_module (shared)
 vhost_alias_module (shared)
 watchdog_module (shared)
 dav_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 lua_module (shared)
 mpm_event_module (shared)
 proxy_module (shared)
 lbmethod_bybusyness_module (shared)
 lbmethod_byrequests_module (shared)
 lbmethod_bytraffic_module (shared)
 lbmethod_heartbeat_module (shared)
 proxy_ajp_module (shared)
 proxy_balancer_module (shared)
 proxy_connect_module (shared)
 proxy_express_module (shared)
 proxy_fcgi_module (shared)
 proxy_fdpass_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_hcheck_module (shared)
 proxy_scgi_module (shared)
 proxy_uwsgi_module (shared)
 proxy_wstunnel_module (shared)
 ssl_module (shared)
 systemd_module (shared)
 cgid_module (shared)
 http2_module (shared)
 proxy_http2_module (shared)
6

eventually ? what gives
time httpd -t -D DUMP_MODULES >/dev/null

7

real 0m12.403s
user 0m0.020s
sys 0m0.026s

I may have identified the cause of the problem, but I am at a total loss to explain it: both UDP and TCP IPv4 DNS is being completely and totally blocked.

8

Ah.

https://dnssec-analyzer.verisignlabs.com/ doesn’t support IPv6 at all.

Using a validating, dual-stack resolver, I can resolve the domain, but it’s slow.

http://dnsviz.net/d/corequery.uk/dnssec/ is dual-stack, and it confirms that it can’t access the nameservers over IPv4 UDP – I’m not sure if it tried TCP – and it says the DNSSEC configuration is valid.

(Let’s Encrypt’s resolvers are dual-stack too.)

Edit: I reloaded DNSViz and it said UDP was down over IPv4 and IPv6.

9

Thanks, - so it appears that my DNS and DNSSEC, both, are good; but that I have a serious protocol issue that I cannot resolve.

I have ruled out any firewall issues, also resolv.conf problems, and can confirm that my named.conf and zones are valid, so I am not sure what else I can do with this.

I am going to re-install the distro, after which I will just avoid Fedora 30 and find something without borked networking if that does not work. I could raise the issue with the Fedora community; but technical issues are not their strong point and I would likely just to be told to check my (not installed) firewalld.

10

Here is a link to another test result. It shows problems with all IPV4
About your original problem, 12 seconds to dump modules don’t seem normal at all since an old container on a Raspberry PI with storage on SD card don’t come over 4 seconds and that’s the first query, the following comes down to 2 s, and after that it’s below .2 s, granted with less modules but something is not right here. Apache is doing strange things it seems.

11

Thanks, - I, too, use Zonemaster (it is very informative), and what you say confirms my suspicions that there is something seriously wrong with the Fedora 30 distro (it is clean install in case you are wondering), and httpd -t returns Syntax OK; but, yes, very slow.

One other point, too: Apache would surely not be responsible for IPv4 TCP and UDP connections being blocked, so what could be causing that, if not a firewall or named?

12

Apache can’t be responsible for your DNS server being slow (from what I can see one of your DNS servers, ns2, is the computer where Apache and certbot run) but maybe in the process of parsing the config Apache could do some DNS queries. I can’t imagine why but stranger things are being done by programmers.

13

Well whatever was causing the problem is still a mystery, but I reinstalled the distro and it not only resolved the problem but is now running about 5 times faster; so the problem is definitely now resolved and I appreciate the time and effort that everyone, including yourself, have put into helping me with this, - thanks.

2 Likes
14

For problems like this in the future, you could consider using strace or ltrace to see what the slow function calls are—which could confirm whether the problem is related to a DNS lookup or something else.

15

Thanks, - noted, - it certainly would have been interesting to know what it was that caused the problem.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.