Flying above 400 feet is only a problem because aircraft can fly down at 500 feet. Do we really want aircraft at only 500 feet? I wouldn't want even a typical general aviation aircraft going over my house at that height. Make it an Airbus 380 and it might be entertaining.
The solution is to raise the height for normal aircraft. That makes room for drones. We could give the drones from 500 to 1900 feet, putting normal aircraft at 2000 feet and above.
There already are restrictions about how low and where you can fly a 'real' airplane. There is even a regulation that would keep an airplane from flying over your house at 500 ft. under most circumstances. The reality is that most pilots don't want to be 500ft above terrain unless they are VERY close to the runway
There are also plenty of good reasons to be under 2,000 ft. The most general is that all training and approach for VFR ops at airports is done at 1,000 ft. by default.
Airspace and altitudes are pretty complex, for a more involved example: A VFR flight departing North from Boeing field in Seattle HAS to stay below 1800 ft but above the highest parts of the city at ~750 ft. Because the air-space above 1800 belongs to jets on instrument approach to Sea-tac. In other words, to get out of that airfield you have to fly underneath the 747s. Seattle is just one example, there's weird airspace like this that balances the needs of different users just about everywhere.
For what it's worth, in any urban area, the 1000 feet above highest object within 2000 feet horizontally is much more likely to apply. While the actual regulations are somewhat loose on definitions, most pilots I know would describe the "500 feet from anything" fallback regulation as being applicable only when there are no structures or other populated places nearby.
In any case, raising floor altitude for non-RC/unmanned aircraft to 2k feet would pose fairly substantial challenges as, even ignoring the fact that airports require approaches below 2k feet and are often near populated areas, lots of commercial applications for light aircraft require low altitudes. Survey, photography, news gathering, law enforcement, sightseeing, etc. These are many of the same applications drones are posed for... there doesn't really seem to be any way to simply separate these types of traffic entirely.
The solution is to raise the height for normal aircraft. That makes room for drones. We could give the drones from 500 to 1900 feet, putting normal aircraft at 2000 feet and above.