Ya there are definitely many ways of looking at this. I found the link about rotor power halving for each doubling of diameter (where it says: "So in this process the power is halved to (2m)(v/2)2/2 = mv2/(2x2) = P/2."):
Downwash definitely figures into this, as well as using fewer blades.
But, the key point is that we intuitively know that smaller craft require exponentially less power than larger craft. We could build a large craft with a very wide rotor to approach the efficiency of small craft, but I think it will turn out to be easier to use multiple small craft and scale how much weight we lift linearly.
"the key point is that we intuitively know that smaller craft require exponentially less power than larger craft."
This is where your intuition has let you down. Think about a normal helicopter's lifting capability and range as you go between models and full size aircraft. Or why you see bees walking.
http://www.pitt.edu/~jdnorton/Goodies/TE-antiTE/
Downwash definitely figures into this, as well as using fewer blades.
But, the key point is that we intuitively know that smaller craft require exponentially less power than larger craft. We could build a large craft with a very wide rotor to approach the efficiency of small craft, but I think it will turn out to be easier to use multiple small craft and scale how much weight we lift linearly.