I scrape lots of data from services while logged in. Scraping is done with curl,... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		ikeboy on June 5, 2018 \| parent \| context \| favorite \| on: Observations running 2M headless sessions I scrape lots of data from services while logged in. Scraping is done with curl, but every so often cookies expire, so I use selenium to login, save cookies, then go back to curl.

smallbigfish on June 5, 2018 [–]

Isn't curl fit to also do the login part and save the cookies?

ikeboy on June 5, 2018 | [–]

Tons of stuff get loaded during login, hard to go through each one and code the proper urls and POST params etc, quicker to write login code.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact