As a owner of a mid range Android phone with a phone jack, I had the same opinion. But now I see there's more to it, the quality of the DAC makes a huge difference.
Interesting - which devices have you found to be good and bad, and how does the difference manifest?
I tend to notice drop offs for sampling bit rate, and can hear the difference with Bluetooth, but I've personally never connected to any device playing decent quality streams/downloads and noticed a difference. Not saying no-one can notice it, just find it interesting I've never done so!
FWIW I listen through some AKG 550s, which I can highly recommend.