At the risk of stating the obvious:

sudo apt-get autoremove unity && sudo apt-get install [xubuntu-desktop|kubuntu-desktop|lubuntu-desktop|etc.]

Or just install the one you want in the first place: http://cdimage.ubuntu.com/ has images for all eight official Ubuntu flavours.

Seriously, it can't hurt you just by being in the repositories if you don't install it...

I was very sceptical of Unity at first. When it was originally introduced in 11.04, it was obviously not ready for prime time. Unity is much more polished now. Unity used to be a resource hog and all kinds of glitchy. Now, Unity is efficient, stable, well thought out, and quite usable. I prefer it to Gnome Shell, KDE 4x, XFCE, Gnome 2, Mate, Cinnamon, and even Xmonad. I don't understand the disdain for Unity these days except for maybe the Amazon crap (which can be disabled).

It's not as bad as it used to be, actually. It used to be a deal breaker for me, but now even though I still don't like it, I don't mind it either.