Hi all,
I'm trying to crawl a site ( GLO) that has an age gateway that blocks screaming from to crawl the pages, except of the homepage.
I found a solution by adding in the Configuration > HTTP Header > User-Agent the age but there are no details how actually make this configuration.
Does anyone knows exactly how to do it?
Thanks,
George
[link] [comments]
from Search Engine Optimization: The Latest SEO News https://www.reddit.com/r/SEO/comments/hypasx/screaming_frog_crawl_with_age_gateway/>
No comments:
Post a Comment