The people who aren't dependent and don't qualify for government provided dope. And there could be a lot of such people: imagine how much less risky a heroin habit sounds when you know that as as an addict you can get free clean drugs reliably, and that it will always be in front of a drug expert.
And it is less risky. I'd support a system that gives I'd more junkies of the junkies aren't destroying their health, aren't stealing, have jobs.
But of you're only allowing heroin use for dependent users in a clinic, you won't destroy the black market. There are still people developing new heroin addictions in Switzerland. And not only new users will use the black market to get heroin. Anyone who wants to use it in their own home, anyone who wants to keep their use secret will have no other choice.
Hassle? Stigma? Hardcore drug addicts will do anything to get their fix. I have known heroin addicts, for example (one was a friend who sadly got sucked into that world). When you're zonked out on heroin most of the time, the way society perceives you is of very, very little consequence.
As another commenter mentioned, I'd imagine a lot of drug trade (maybe not in heroin, but definitely in cocaine) is not hardcore drug addicts, but occasional (or even regular) users who live (relatively) normal lives.