A few people at either extreme wouldn't make the median go anywhere, that's the whole idea of using median instead of arithmetic mean. Probably this explains the difference between genders: say three women have 1,5 and 9 partners, while three men have 0,0 and 15 partners, the means are 5 in both cases, but the medians 5 vs. 0.