It's an iterative process of tweaking, then stepping back to take a wider view, then tweaking some more, then stepping back again to evaluate, etc... until either (1) you're intuitively happy with the result or (2) your result conforms to some objective standard, or (3) both. It requires introspection and knowing how to "hack" your intuition. Different people hack their intuition in different ways, here is one way to do it, one that doesn't require drugs and going on a trip to India to meet holy men: http://news.ycombinator.com/item?id=3175272
Some people have this naive romantic notion that you basically go through some period of "suffering" as an artist and then suddenly you have some "grand vision" unfolding in front of you that's hidden to everybody else. What they miss is all the boring, often tweaking-like work that you have to perform before you are faced with that vision. Suffering without doing work doesn't lead to anything.
Sorry that Gladwell has sullied your opinion of writers... but all the good writers (emphasis on "good") I've known and currently know are insanely good at the creative process and lead the way in creative development.
It's the copywriter who produces the creative concepts in an advertising agency. Those commercials you see and those taglines/slogans you remember? Those were the brainchild of a copywriter sitting at a desk somewhere, mulling over ideas, soaking up culture and transforming it into a memorable view of that culture. The copywriter imagines. Then documents. Then edits. Then reimagines. Then lets a bunch of others give their opinions. Then edits some more.
Not sure what writers you've come across, but you might want to ease up on making sweeping statements about all - or even 80% of writers. Oh, and most working writers -- the ones who aren't starving -- are hard workers who don't believe in climbing the mystic mountain to seek out the Muse, as you describe.
Some people have this naive romantic notion that you basically go through some period of "suffering" as an artist and then suddenly you have some "grand vision" unfolding in front of you that's hidden to everybody else. What they miss is all the boring, often tweaking-like work that you have to perform before you are faced with that vision. Suffering without doing work doesn't lead to anything.