Monday, August 3, 2020

I need some help with my robots.txt

Hi Everyone,

For some reason I keep getting a Robots.txt issue on web.dev , however, the site doesn't tell you what's wrong. I'm not getting any issues out of SEMrush either. So I'm not sure what I'm doing wrong here. Can anyone tell me what might be wrong about our Robots.txt? If anyone spots a mistake, I'd be very happy to hear it. Thanks a bunch in advance!

Here it is:

User-agent: *

Disallow: /wp-

Disallow: /feed

Disallow: /feed/

Disallow: /author

Disallow: /cgi-bin/

Disallow: /wp-admin/

Disallow: /comment/

Disallow: /comments/

Disallow: /trackback/

Disallow: /wp-includes/

Disallow: /wp-login.php

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.\*\*\*\*\*\*\*\*\*.nl/sitemap\_index.xml

User-Agent: Googlebot

Allow: .js

Allow: .css

submitted by /u/Doctor_Philly
[link] [comments]

from Search Engine Optimization: The Latest SEO News https://www.reddit.com/r/SEO/comments/i2td9k/i_need_some_help_with_my_robotstxt/>

No comments:

Post a Comment