Hi Everyone,
For some reason I keep getting a Robots.txt issue on web.dev , however, the site doesn't tell you what's wrong. I'm not getting any issues out of SEMrush either. So I'm not sure what I'm doing wrong here. Can anyone tell me what might be wrong about our Robots.txt? If anyone spots a mistake, I'd be very happy to hear it. Thanks a bunch in advance!
Here it is:
User-agent: *
Disallow: /wp-
Disallow: /feed
Disallow: /feed/
Disallow: /author
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /comment/
Disallow: /comments/
Disallow: /trackback/
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.\*\*\*\*\*\*\*\*\*.nl/sitemap\_index.xml
User-Agent: Googlebot
Allow: .js
Allow: .css
[link] [comments]
from Search Engine Optimization: The Latest SEO News https://www.reddit.com/r/SEO/comments/i2td9k/i_need_some_help_with_my_robotstxt/>
No comments:
Post a Comment