A single sidewinder is almost 200 pounds and GBU-31/32/38 JDAMs are 500-2,000 pounds each, so accounting for the pilot and ejection seat, you're only getting a small bomb's worth of weight.

Making them smaller significantly reduces the volume available for weapons, sensors, and fuel too much IMO. See the Predator, which doesn't have a large weapons loadout and still has the cockpit dome for the radar.