Monday, September 30, 2019

How do I stop semrush bot from crawling add-to-cart links?

How do I stop semrush bot (and other bots) from crawling add-to-cart links?

I tried adding the following dissallow command to my robot.txt file, but the bot is ignoring it.

Dissallow: /?add-to-cart* 
submitted by /u/Mesmer7
[link] [comments]

from Search Engine Optimization: The Latest SEO News https://www.reddit.com/r/SEO/comments/dbowy7/how_do_i_stop_semrush_bot_from_crawling_addtocart/>

No comments:

Post a Comment