Why not? I certainly don't want to be driven by somebody who doesn't want to drive me and hates my guts for being forced into driving me. I'd rather he refused and another willing driver pick up the job.
One would think the price would increase until someone would pick you up.
Uber should add the ability to bid up your route--like when it's taking a long time to match you to a driver, or you want to entice a driver who is nearby so you can get where you're going sooner.