Think of it this way: a barrel shifter has to be able to "carry" every bit to (potentially) every other bit.

Yep, but also while aligned. Bits x and y are going to be the same distance apart, always, unless shifted off the end (where they can wrap or be lost). Thinking of this in the pub it seemed a butterfly thingy would be appropriate <https://en.wikipedia.org/wiki/Butterfly_network>.

Further thinking suggested there'd be a ton of wires doing this, and perhaps it's the wiring that's taking up the silicon?

It's the multiplexers. Also, critical path is going towards limiting your speed: https://en.wikipedia.org/wiki/Barrel_shifter