That is why in most countries (I can't speak for the US) they cap the maximum withdraw limit to something like $100. That way the risk to the bank and consumer is minimal.

Here in Australia our largest bank (Commonwealth Bank) has an app that will generate a code that you can enter into the ATM to make a withdrawal i.e. card not required. It has a similar risk profile to NFC.