The downside is that you have declining birth rates and a slowing economy. Despite all the non-sense from the current admin, you do actually need immigrants. Trump has said so himself as his admin keeps screwing things up (see the Hyundai fiasco). The average American citizen has no idea about how immigration in this country works. People are overwhelmingly in favor of legal and skills based immigration.

>but congress taking action I think is very likely to screw things more than they currently are

Congress' inaction and paralysis is the reason we find ourselves in this morass. Congress has abdicated its responsibility and ceded power to the executive.