> I think the browser API of WebRTC requires the server to know the client's certificate as well
Yes, as it stands, the browser WebRTC API is designed around the idea that a p2p connection is being bootstrapped by connecting two of the leaves of a star topology to each-other. For this use-case, it's a pure optimization: the server starts off intermediating between peers; then at some point, the peers each figure out that the other one can do WebRTC, so the server introduces the peers to one-another (gives each peer the other peer's authoritative DTLS certificate hash), and the peers stop talking to the server and start talking to one-another instead.
I don't think this is any kind of equilibrium state, though. This is the way things are because because WebRTC evolved purely as an optimization for companies like Google and Facebook to avoid needing to deal with massive streaming-media traffic fan-in. In their cases, they already had multimedia chat platforms that used authoritative central backend servers intermediating media streams; and they just wanted to introduce a way for a browser to "upgrade" from talking to the backend, to talking to another browser. Like upgrading an HTTP flow into a Websocket flow.
But a lot of the work since then, on Data Channels, DTLS and so forth, has been at the behest of other companies (e.g. Ericsson) whose motivation isn't to decrease load on their massive video-chat backends, but rather to do other things, like enabling browsers to talk directly to e.g softphones or IP cameras, with both sides throwing packets directly at one-another without some kind of gateway in the way.
I fully expect that while you might not get browser STUN/TURN nodes in the near future, we will at least see something like browsers offering the option of mutual authentication of DTLS certificates through a specified X.509 CA set (where both peers have been issued client certificates signed by one of the CAs trusted in the set; and where the CA could very well be a self-signed private one unique to the service.)
Do we even need "mutual authentication of DTLS certificates", though? I would think it should be sufficient for either the server's certificate to be CA-signed or -- even easier and even better as far as I am concerned (as to me the CA system is itself bowing to an authoritative system I would rather avoid) -- for the client to just already know the certificate hash of the server (as part of its address).
The only problem I am seeing is that I want to be able to leave the peer certificate hash null and just authenticate with anyone on the other side. Like, I actively don't want "mutual authentication of DTLS certificates" for when a user first connects. For later re-connections it might be useful for the client to leave the network a certificate hash, but for that initial connection I want one side to not have a certificate, same as using TLS to connect to any known HTTPS server.
> Do we even need "mutual authentication of DTLS certificates", though? I would think it should be sufficient for either the server's certificate to be CA-signed
I mean, we're talking about something where the core use-case is for equal peers to be connecting to one-another, where who is dialing whom is entirely arbitrary. While your app on top of WebRTC might have nodes that do server-like things and other nodes that don't, on the WebRTC level, there is no "server"; for any given WebRTC session, either one of the peers might end up doing the dialing or the listening, perhaps with this even changing for the same pair of peers on a connection-by-connection basis.
In such a context, mutual auth is the only kind of auth that makes sense to me. Both sides establish a stable identity (beyond their relatively-unstable current IP address), so that each side's User Agent has at least one piece of reliable information it can use to whitelist or blacklist other nodes' connection attempts by.
Consider: we're struggling right now to get the phone network to adopt protocols that allow each side of a phone call to non-repudiatably identify the entity on the other end of the call (i.e. for it to be impossible for each side to fake its phone number.)
So why would we want to introduce a new protocol in the year 2024, supposedly for p2p connectivity, where at least one side of the connection is allowed to be the equivalent of the phone network's "Unknown Caller"?
Or, to put that another way: do you really want to enable your browser to be DDoSed? By other browsers, who've landed on some malicious webpage that runs in a background tab, opening never-ending WebRTC connections to target hosts according to the websocket-delivered instructions of some booter service?
Mutual auth at least ensures that connection attempts like those will be dropped at DTLS establishment time. To talk to a WebRTC session established by a page from origin X running in your browser, those tabs in those other people's browsers will need access to DTLS certs that are signed by a party trusted by the code in your browser (i.e. the code written by origin X.) And those DTLS certs would in turn presumably only be made available to... pages from origin X. (Or its logistical data partners.)
But, I don't want "origin X", I want "peer-to-peer". I don't want some server somewhere providing the authorization to enter the network by gating the CA: I want a mutually interoperating set of web applications written by different developers all able to run code (maybe in a library) that bootstrap themselves into a permissionless peer-to-peer network. As for your concern about having to deal with a ton of incoming connections, such is the life of a supernode: if you aren't up for that, don't sign up to be a supernode. But like, without this, why do we care about "supernodes" anyway? If you have some centralized server you have to talk to to get certificates, just have it do signaling as well! :(
Yes, as it stands, the browser WebRTC API is designed around the idea that a p2p connection is being bootstrapped by connecting two of the leaves of a star topology to each-other. For this use-case, it's a pure optimization: the server starts off intermediating between peers; then at some point, the peers each figure out that the other one can do WebRTC, so the server introduces the peers to one-another (gives each peer the other peer's authoritative DTLS certificate hash), and the peers stop talking to the server and start talking to one-another instead.
I don't think this is any kind of equilibrium state, though. This is the way things are because because WebRTC evolved purely as an optimization for companies like Google and Facebook to avoid needing to deal with massive streaming-media traffic fan-in. In their cases, they already had multimedia chat platforms that used authoritative central backend servers intermediating media streams; and they just wanted to introduce a way for a browser to "upgrade" from talking to the backend, to talking to another browser. Like upgrading an HTTP flow into a Websocket flow.
But a lot of the work since then, on Data Channels, DTLS and so forth, has been at the behest of other companies (e.g. Ericsson) whose motivation isn't to decrease load on their massive video-chat backends, but rather to do other things, like enabling browsers to talk directly to e.g softphones or IP cameras, with both sides throwing packets directly at one-another without some kind of gateway in the way.
I fully expect that while you might not get browser STUN/TURN nodes in the near future, we will at least see something like browsers offering the option of mutual authentication of DTLS certificates through a specified X.509 CA set (where both peers have been issued client certificates signed by one of the CAs trusted in the set; and where the CA could very well be a self-signed private one unique to the service.)