I use this > http://testico.net/ then i test it on my device (iphone 4S) where i relocate it in a couple of pages. I also put it inside a folder too to check if the icon is 'readable' there. Then i ask a few people if it stands out. Then sometimes, i sleep then check it first thing in the morning where i am groggy. If it passes all those then its a good icon. I may have to add a check when I am in a bad mood or hungry for my method to be ultra comprehensive.
If you have budget to spare, A/B test a few different variants on the Facebook ad platform.
Would it kill the people at testico.net to display a one-sentence description of what their site does? I had to view HTML source and read the meta description:
"You can to put the icon you've created on your iPhone or iPad home screen, and generate automatic previews of standard icon sizes on the standard iPhone and iPad screens"
That really doesn't stand out in the App Store listings- the detail work is all to emphasize the 512x512 version that doesn't matter. The only color is in the middle keyhole, which isn't particularly visible at the smaller resolution.
The old glowing keyhole was dramatically better- OP might want to take a look at merging the two, with the new box design and the old glow, to help this stand out more.
Yeah, the old icon with the bright glow really stuck out to me as well. Wish we could have seen A/B tests, even ones off web ads like someone recommended would have been better than nothing.
This is the most insightful post on HN about icons that has ever been submitted. A lot the people here would do well to take notice and learn something about art design.
One of the takeaways of the article is that the app icon is a form of a sales pitch to a potential user. You can have a great product, but a poor icon will hinder the visibility of said product. That being said, if I had to choose between a great app icon or a great product, I'd choose the latter. But I'm not convinced you are forced to choose only one.
If you have budget to spare, A/B test a few different variants on the Facebook ad platform.