Here's the abstract for a nice review of search engine law: http://works.bepress.com/james_grimmelmann/13/ If I remember correctly, indexing a site that asks to not be indexed might be illegal as an illegal tresspass, but it is not settled law. The argument is that you are stealing resources (computer time) from the site owner.
There are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies.
The two that jump to mind are Authority Labs and SEOmoz.
IANAL but AFAIK it is only a civil matter (i.e. not illegal) since it is a usually prosecuted as a tort of trespass to chattels. For such a case to succeed the prosecution needs to show that the actions of the defendant deprived them of use of the good they were trespassing on. i.e. they need to cause enough of a burden on the servers that the claimant or their customers could not use the service.
What I'd care more about is whether the people posting the ads wish to be indexed; I'd be surprised if many don't. (I don't know about the legal bearing either.)
Perhaps craigslist could put up a checkbox (like they do with the never-checked "It's okay to contact me about products ...") that says something like:
"I'm okay with people finding this listing through another service."
Perhaps because they don't want people to get to their listings that way even if they want to, and thus don't want your opinion? (Edit: and perhaps they really aren't as interested as they claim in making it easier for buyers and sellers to find each other?)
