How To Optimize WordPress Robots Txt For SEO


How to Optimize WordPress Robots txt for SEO is the latest 2016 video tutorial for adding robots.txt file for WordPress sites
Googlebot and other user-agents obey the directives specified in robots.txt file. These directives when used for SEO ‘Search Engine Optimization’ can be of use when you want your WordPress site to be crawled more efficiently by Google. To learn more about robots.txt files visit:

You can also use Google Search Console to test your current robots.txt file directives to see which (if not all) user-agents (web crawlers) that Google uses can access your site by using this link:

Here’s a video lesson by RankYa that shows you how to use robots.txt tester:

Other user-agents Google uses:

User-agent: Googlebot
Disallow:
Allow:
User-agent: Googlebot-Mobile
Disallow:
Allow:
User-agent: Googlebot-Image
Disallow:
Allow:
User-agent: Googlebot-News
Disallow:
Allow:
User-agent: Googlebot-Video
Disallow:
Allow:
User-agent: Mediapartners-Google
Disallow:
Allow:
User-agent: Adsbot-Google
Disallow:
Allow:

This video lesson for WordPress also shows PHP methods for controlling crawling and indexation for Googlebot.

You can also password protect your WordPress sites’ private folders or files by using password protection

Sample robots.txt file and WordPress directives can be downloaded on this page as shown in the video tutorial

Leave a Reply

Your email address will not be published. Required fields are marked *

Exclusive Discounts on SEO & Passive Income Tools - Up to 90% OFF

Newly Released Cutting Edge Software & Tools

Let's face it, we've all been victim to shinny object syndrome. What if we could sort the wheat from the chaff and provide you insider tips on the best pre-released or just released tools in the industry? We don't want to fill your inbox with crap, just a weekly list of the best new tools available, at AMAZING DISCOUNTS!

We hate spam, so should you. We will never sell or giveaway your details.

Exclusive Discounts on SEO & Passive Income Tools - Up to 90% OFF

Newly Released Cutting Edge Software & Tools

Let's face it, we've all been victim to shinny object syndrome. What if we could sort the wheat from the chaff and provide you insider tips on the best pre-released or just released tools in the industry? We don't want to fill your inbox with crap, just a weekly list of the best new tools available, at AMAZING DISCOUNTS!

We hate spam, so should you. We will never sell or giveaway your details.