There have been efforts to do this. Back during the first crypto-wars I had some code in Java that would give it capabilities similar to the way that the language Joule did it (basically the class loader would elide any methods from the loaded class based on capabilities so they weren't even available) I got a couple of patents out of it but sadly the politics inside of Sun kept it from going anywhere useful.
Highly constrained OSes and languages that do so in order to minimize attack surface tend to be challenging to work in. As a result they constrain productivity which increases time to market and the folks who got something out, even insecure, would "win" the market. It was a sad thing to see happen.
I found that working on Green Hills' Integrity was quite a bit easier than working in user space on a typical UNIX userspace. The primary reason is that the API for Integrity was designed in the late 90's, where the POSIX API was inherited from the first thing folks thought of in the late 70's.
The other aspect was that IPC via message passing is a very natural way to program.
SeL4 is a small microkernel, not a complete operating system. It is very, very cool, and deserves more adoption, but a customer would need a load of stuff on top of it for it to be a viable option.
