Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This. Bytedance's official spider has a clear User-Agent tagged Bytespider, but OP didn't mention what they're seeing.


This isn't spider traffic though, the traffic pattern indicates that it's a special-purpose bot designed to hit Cobalts internal API in particular. A generic spider probably wouldn't even be able to find the API endpoints that are only referenced by Javascript, nevermind consistently hit the API with a valid video URL from a residential proxy then switch to a different IP address to download the result every time.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: