I would just disable down-voting on touch screen devices. At the expense of some functionality you avoid the issue of hiding valuable content.