# What is the current status of the implementation of multi-viewpoint validation?

**URL:** <https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291>\
**Category:** Issuance Tech\
**Created:** [December 10, 2019, 9:14am UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291 "2019-12-10T09:14:10Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![lipflip010](https://avatars.discourse-cdn.com/v4/letter/l/a698b9/32.png) [@lipflip010](https://community.letsencrypt.org/u/lipflip010)\
**Post date:** [December 10, 2019, 9:14am UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/1 "2019-12-10T09:14:10Z")

</div>

I’m researching for countermeasures against BGP Attacks as discussed in this paper presented at USENIX: [Bamboozling Certificate Authorities with BGP](https://www.usenix.org/system/files/conference/usenixsecurity18/sec18-birge-lee.pdf)

So far I only found this post in the Let’s Engrypt Forum: [Using BGP to Acquire Bogus TLS Certificates](https://community.letsencrypt.org/t/using-bgp-to-acquire-bogus-tls-certificates/38627)

I was also trying to find information in the GitHub repo of Boulder but couldn’t really find my way around.

---

<div class="post-metadata">

**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\
**Post date:** [December 10, 2019, 9:25am UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/2 "2019-12-10T09:25:37Z")

</div>

This isn’t an authoritative answer, but from what I have observed recently, both staging and production now have multi-VA enabled, but ~~production~~ they are both configured to only require a single VA to get a successful response.

---

<div class="post-metadata">

**Author:** ![orangepizza](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/orangepizza/32/19597_2.png) [@orangepizza](https://community.letsencrypt.org/u/orangepizza)\
**Post date:** [December 10, 2019, 10:06am UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/3 "2019-12-10T10:06:35Z")

</div>

Does DNS challenge(or just DNS lookup) also do multi-VA thing?

---

<div class="post-metadata">

**Author:** ![cpu](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/cpu/32/84514_2.png) [@cpu](https://community.letsencrypt.org/u/cpu)\
**Post date:** [December 10, 2019, 1:26pm UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/4 "2019-12-10T13:26:28Z")

</div>

@lipflip010 Thanks for the question,

We are nearing a production enforcing deployment of multi perspective validation but haven't launched it yet. I'm hopeful we can do this in Q1 2020 but it's primarily blocked on SRE time.

> [@\_az](#):
>
> they are both configured to only require a single VA to get a successful response.

How did you test this? My understanding is that the staging environment **is** enforcing multiple perspective validation. It should require 2 of the 3 remote perspectives to agree with the primary perspective.

Production is making the remote perspective validation requests but we are only aggregating the results for analysis and not blocking issuance if they disagree with the primary perspective.

> [@orangepizza](#):
>
> Does DNS challenge(or just DNS lookup) also do multi-VA thing?

All of the challenge types as well as the associated DNS lookups (A, AAAA, CAA, etc) are done by the individual perspectives.

---

<div class="post-metadata">

**Author:** ![Phil](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/phil/32/76801_2.png) [@Phil](https://community.letsencrypt.org/u/Phil)\
**Post date:** [December 10, 2019, 3:49pm UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/5 "2019-12-10T15:49:26Z")

</div>

> [@cpu](#):
>
> I’m hopeful we can do this in Q1 2020 but it’s primarily blocked on SRE time.

I'm hopeful for this too. From the SRE perspective (knee-slap), we're developing capacity plans in preparation for a hardware refresh to support future growth.

---

<div class="post-metadata">

**Author:** ![cpu](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/cpu/32/84514_2.png) [@cpu](https://community.letsencrypt.org/u/cpu)\
**Post date:** [December 10, 2019, 9:14pm UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/6 "2019-12-10T21:14:08Z")

</div>

Had a few more thoughts about this thread at the end of the day 🙂

> [@lipflip010](#):
>
> I’m researching for countermeasures against BGP Attacks

We'd love to hear about your research/findings. We try to keep an eye on this space and support academic research where we can. If you have a project relevant to us or security research you're performing in this space please feel to reach out in forum DM or via e-mail if you think there's something we can do to make Let's Encrypt's domain validation more secure.

> [@lipflip010](#):
>
> I was also trying to find information in the GitHub repo of Boulder but couldn’t really find my way around.

If you have specific questions about what implementation details you'd be interested in reading I can provide some pointers. We haven't done a great job of documenting the high level system design, sorry about that! I'll try to land some documentation in the new year (Filed [#4609](https://github.com/letsencrypt/boulder/issues/4609) to keep myself honest).

If you follow the [Development Instructions](https://github.com/letsencrypt/boulder#development) to set up a Boulder environment in Docker and then change your `docker-compose.yml`'s `BOULDER_CONFIG_DIR` to `test/config-next` instead of `test/config` you'll have a Boulder environment configured with two primary VA instances (validation requests are load balanced across the two) and two remote VA instances (each primary VA will ask both remote VAs to perform matching validations for each primary validation). Of course this is a dev env so both the primary and remote VAs are all running on one host.

The primary and remote VAs are both the same piece of software, the `boulder-va` service ([cmd here](https://github.com/letsencrypt/boulder/tree/master/cmd/boulder-va), [package here](https://github.com/letsencrypt/boulder/tree/master/va)). The boulder-ra uses [the same RPC interface](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/va/proto/va.proto#L8-L10) to ask for a primary validation as the primary VA uses to ask a remote VA for a confirmation validation.

Primary VA instances know they are a primary based on the presence of the `"remoteVAs"` configuration element. If present it specifies gRPC service addresses for other VA instances to use as remotes. There's also a handful of feature flags that control how the primary VAs handle the remote VAs.

In the dev env with `config-next` the two primary VAs are `va1.boulder:9092` and `va2.boulder:9092` and use [`test/config-next/va.json`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va.json) as their configuration. This config file specifies two `"remoteVA"s`, `va1.boulder:9097` and `va2.boulder:9098` and enforces [that a maximum of 1 of the 2 remote VAs disagree](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va.json#L44) with the primary VA for all validations. The remote VA instances use [`test/config-next/va-remote-a.json`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va-remote-a.json) and [`test/config-next/va-remote-b.json`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va-remote-b.json) as their config files.

There's some integration tests that test this end to end. The most relevant is probably [`test_http_multiva_threshold_fail`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/v2_integration.py#L876-L908). It tests that a HTTP-01 challenge made to a webserver that only gives the correct key authorization to the primary VA and not the remotes will fail the multi-perspective validation.

Hope that helps give you a place to start. I'll be out of the office for a ~week but will follow-up if you have more questions when I'm back.

---

<div class="post-metadata">

**Author:** ![lipflip010](https://avatars.discourse-cdn.com/v4/letter/l/a698b9/32.png) [@lipflip010](https://community.letsencrypt.org/u/lipflip010)\
**Post date:** [December 11, 2019, 2:52pm UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/7 "2019-12-11T14:52:32Z")

</div>

First of all, thanks to all of you for your detailed responses!

> [@cpu](#):
>
> We’d love to hear about your research/findings. We try to keep an eye on this space and support academic research where we can. If you have a project relevant to us or security research you’re performing in this space please feel to reach out [...]

I think I have to clarify the kind of research I did 😅 I'm in my undergrad now and had to prepare a presentation based on the paper I mentioned in my question. Towards the end of preparition I thought it might interest the audience which countermeasures are already implemented and to what degree. So I'm not really doing an extensive research.

> [@cpu](#):
>
> If you follow the [Development Instructions](https://github.com/letsencrypt/boulder#development) to set up a Boulder environment in Docker [...]

Thank you for your effort to provide me instructions for running and testing it myself; I feel honored by that, though I think this goes beyond the scope of my presentation.

---

<div class="post-metadata">

**Author:** ![cpu](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/cpu/32/84514_2.png) [@cpu](https://community.letsencrypt.org/u/cpu)\
**Post date:** [December 18, 2019, 3:55pm UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/8 "2019-12-18T15:55:22Z")

</div>

> [@lipflip010](#):
>
> First of all, thanks to all of you for your detailed responses!

Happy to help!

> [@lipflip010](#):
>
> I think I have to clarify the kind of research I did 😅 I’m in my undergrad now and had to prepare a presentation based on the paper I mentioned in my question. Towards the end of preparition I thought it might interest the audience which countermeasures are already implemented and to what degree. So I’m not really doing an extensive research.

Thanks for clarifying. I hope your presentation was successful 🙂

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/letsencrypt/original/3X/c/a/ca6c06ea1ea201324bba7048c6841ce60236468d.png) [@system](https://community.letsencrypt.org/u/system)\
**Post date:** [January 17, 2020, 3:55pm UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/9 "2020-01-17T15:55:24Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![jsha](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/jsha/32/12_2.png) [@jsha](https://community.letsencrypt.org/u/jsha)\
**Post date:** [February 3, 2020, 11:45pm UTC](https://community.letsencrypt.org/t/what-is-the-current-status-of-the-implementation-of-multi-viewpoint-validation/108291/10 "2020-02-03T23:45:06Z")

</div>

FYI we’ve announced the launch plan for multi-viewpoint validation: [ACME v1/v2: Validating challenges from multiple network vantage points](https://community.letsencrypt.org/t/acme-v1-v2-validating-challenges-from-multiple-network-vantage-points/112253).
