If you setup your dotfiles correctly, your vim environment should be accessible with a simple `git pull`.
The plugins are very worth it, there's no need to handicap yourself locally just so your environment matches some bare bones server config.
And regardless, the basic vim motions are always there - you don't lose much when just editing some config files on a server even if you have to use vanilla vim
My approach is to just use all the tools. I use clumsy IDEs when I really need powerful debugging. I use vim when I need powerful editing/macros. Sometimes I do code browsing with ctags. Sometimes I do code browsing with Eclipse. There's no need to be a purist.
Except vim. No plugins into vim. That is sacrilege.