It's not so interesting to me that people continue to feel the need to build a scraping framework (it's an excellent beginner project because it encompasses so much of web development) but why HackerNews finds scraping frameworks so interesting. There seems to be a scraping framework at the top of HN every week or two.

Personal theory, as someone who has done a good bit of scraping over the last decade:

Extremely common problem space, with a lot of tantalizing opportunities for a "platform" or "shared language" where one doesn't exist. I see existing tooling like BeautifulSoup, Scrapy, Selenium, as isomorphisms to the "near hardware level" of our problem-space-tool-stack, whereas the abstractions and higher level logic is often defined per the use case.

On top of (or perhaps because of) that, one often writes a lot of boilerplate, but when it comes to genericize, one often ends up writing the tool that genericizes within their problem space, and for all the tantilizing opportunities, the number of "not quite fully intersecting scraping problem spaces" (and associated tradeoffs/different paradigms) is far more massive than I considered when I started any of my own scraping tools.

This has lead me to take a very opinionated view with my own tooling, wherein I build for _ONE SPECIFIC RECURRING SCRAPING PRIMITIVE_. (in my case, treating the whole world as a stream of events. You want something that can be more or less massaged into that? Cool; maybe something for you. If not? Probably want to look at a different set of tools)