Speculation Rules Script & Googlebot

  • farnely

    (@farnely)


    3 days, 9 hours ago

    I recently received a notification from Google that they had found reasons preventing pages from being indexed. (Bear with me; this IS about WordPress). When I checked the Search Console, the following URLs are listed as returning 404s:-

    /wp-*.php, /wp-admin/*, /wp-content/uploads/*, /wp-content/*, /wp-content/plugins/*, /wp-content/themes/my-theme/*, /wp-content/themes/my-theme/*

    They’re not “valid” URLs but they match EXACTLY the list used in the speculation rules script loaded by WP. That’s not a coincidence IMHO.

    Normally, when pages are not being indexed, I check my settings for the affected URLs, make any required changes then mark as fixed in Search Console. I don’t see how I can do that in this case so can someone please explain what I can do to stop Google crawling these URLs.

Viewing 1 replies (of 1 total)
  • Moderator threadi

    (@threadi)

    3 days, 7 hours ago

    The paths mentioned belong to the WordPress core and are not and do not need to be indexed by search engines.

    What does your robots.txt look like? You can use it to exclude these paths.

    Do you use an SEO plugin? That would take care of the appropriate robots.txt entries for you.

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

Tags