"DNS being slow (fuck it, use IPs)" GAAAHH!! <table flip>. The 1980s called, they said youre an f'n idiot. Seriously, this is not "operations". This is "phoning it in" and/or "not understanding the problem".
The problem is DNS resolution isnt cached and potentially has to do a lookup everytime it does a connection. When I was doing that every request (potentially several times), it became expensive. Sounds like I understand it?
That DNS mapped to one node, which will rarely change. If it does, I can spend 30 seconds and deploy a new config, rather than worry about using virtual IPs or anything more complex, let alone with having to configure a cache which would have the exact same problems (delay to change).
Guess what, it's 2012 and the same shit that worked back then works just fine now.
There's a fine line between doing things right, and doing things just to do them.
"The problem is DNS resolution isnt cached" I don't get it. It's like using http withou content-length or content-md5 and complaining about a lack of data integrity.
A DNS cache is trivial, either in your code or a local service. Or hell just use your local gethostbyname(). Your os will probably cache for you. And if it's not you'll pay 0.1ms to hit another box in your dc. But if 30ms every few minutes is still too much use a pref etching resolver like (lib)unbound.
I suppose my real objection is in not Doing The Right Thing. Yes, it may work for your current two box static setup. But small choices contribute a lot of debt that you or someone else has to pay off later. Would be a shame if someone took the message that DNS was just "overhead".
Run DNSMasq locally (as in, same datacenter as the computers that will be using it) and tell it to cache. It's dead-simple to set up. Then point your computers to resolve using it.
You can even add to /etc/hosts and the computers using it as their DNS will resolve it. Depending on how much control you have, DNSMasq will also function as a DHCP server and TFTP server from which you can netboot other servers and do such nifty thing as automatic reinstalls. Useful if you have a separate, internal network and want to set internal IPs, too.
DNSMasq is nice, it's so easy to make up your own, local, dns names.
Do you run several to avoid a single point of failure, or do you just fall back to the "real" dns?
Even with a local DNS server, there has to be some overhead though.. OTOH, avoid premature optimization etc..
So, yeah, ops isn't hard at all, if you don't fucking take the time to do it right.