Even at the time, html scraping was necessary for a lot of things. And lots of services that has open APIs requires access keys. I'm allowed to read your twitter feed in html anonymously, but for json I need to apply for an account in advance and login, for example. Even if we could force open APIs, it's not sufficient.