I think that Chrome allows calling http://localhost from https domain. Other browsers should fix that instead of every application installing their custom certificates into trust store.
Installing trust store certificates for these purposes is asking for trouble. Sure, if the private key is unique per installation it theoretically should be fine, but in reality it can be hard to gather enough entropy to be satisfiably unpredictable, and it's very easy to get this wrong. It's worth noting that there isn't a solid way to limit your CA certificate in the trust-store to the domains you intend to use it with https://security.stackexchange.com/questions/31376/can-i-res...
> in reality it can be hard to gather enough entropy to be satisfiably unpredictable
I think this is a myth, especially on desktop/laptop PCs - a few hundred bytes of entropy at system boot initialize the kernel CSPRNG, which can then generate countless gigabytes of cryptographic-quality randomness on demand.
Well, you need your software to work now, hard to blame you :)
I thought about this problem and there are two workarounds. First workaround is to get agreement with major CA who would allow you to issue a valid certificates for users. So it's like user installs your software, generates private key and you generate signed certificate for that key on your server. I think that plex does that, but it's probably extremely hard and fragile scheme.
Second workaround would be to proxy traffic from your localhost server to your remote server. Remote server would present valid certificate for something like local.yourcompany.com and would decrypt traffic and translate it back to your localhost server. Same with response. So you're doing encryption with remote server and never leak your private key. I'm not sure if CA would be happy with that implementation, but technically I believe it's not a key compromise.
The first workaround also crossed my mind, but it had a couple of drawbacks. First, it required contractual work with a CA and they can easily say: it's not our problem, it is the browser. The amount of time required to set this up would be around a year, maybe more. Also, like you said, it is fragile.
The second workaround I didn't think about. Do you mean we'd change the resolver to resolve 'local.yourcompany.com' to 127.0.0.1 on the local machine? That would work, but would introduce quite some extra latency and add some fragility.
