The overall process is that certificate authorities have to apply to browsers to be listed as trusted roots. The browsers (acting on behalf of their users) decide which certificate authorities they consider trustworthy, through each browser’s activity called a “root program” (as in a course of activity, as opposed to a piece of software).
For example, Mozilla’s is described at
The main thing that a certificate authority has to do in order to apply to be included in a root program is create appropriate policies and infrastructure (like those described in the policy repository that @Osiris linked to) and then be audited through the Webtrust or ETSI certificate authority auditing process to confirm its compliance with its policies.
However, as @Osiris also noted, certificate authorities that directly issue certificates to end users operate as intermediate CAs, which are signed by roots (which may be other organizations, as in the case of IdenTrust signing Let’s Encrypt’s intermediate certificate to approve Let’s Encrypt as a trusted CA). By default, browsers will believe trusted root CAs’ assertions about intermediate CAs’ trustworthiness, so this delegation of trust will be accepted. According to current industry rules, intermediate CAs also have to publish policies and pass through Webtrust or ETSI audits concerning their policy compliance.
Overall, these decisions are made by the web browsers in response to audit statements and also other evidence about CAs’ behavior. For example, browsers have imposed punishments on CAs that they concluded were violating rules or policies or otherwise acting improperly.
You can read about the industry standards that all publicly-trusted CAs’ policies and practices are expected to conform to here: