Sometimes it makes sense to block search engines from indexing your website.
For example, you might buy a brand new domain name and build your WordPress website on a live server to save the hassle of moving it once it’s finished (make sense for beginners), or you might want to redesign an existing site on a temporary subdomain.
In some cases, you might want to build a private website that you don’t want to be found through any search engine. You could be using it for a specific project, an intranet or a private family blog.
If any of these situations apply to you, and you don’t want people to find your site through organic search results, there’s a setting inside WordPress you can use to instruct search engines like Google not to crawl or index your pages.
How to stop search engines indexing your WordPress website or blog
Log into the admin area of your WordPress website and navigate to Settings > Reading and tick/check the box next to Discourage search engines from indexing this site.
Checking this box does two things:
It adds this command to the source code of every page on your site.
<meta name="robots" content="noindex,follow">
These two commands – noindex and follow – tell search bots to not index the page but to follow the links within the page to discover more content. Because this code exists on every page, Google won’t show your pages in the search results.
And it adds this to the robots.txt file
This tells search engine bots to disallow all content. When a bot lands on your domain it checks for a robots.txt file. If it finds one containing the protocols above, it won’t crawl and index your site.
At least, that’s the theory.
The problem with these methods is that you can’t guarantee your pages and other content won’t get indexed.
You MUST remember to change the settings when your site is ready!
When your site is ready for the world to see, you really must go back to the settings page and untick/uncheck the Search Visibility button. If you don’t do that, the search engines won’t index your website.
How to work on your site without blocking search engines
If you want to work on your site and stop people from viewing the pages while it’s in development, but don’t want to block search engines, install a maintenance mode plugin.
There are tons to choose from. Check them put here.
A maintenance mode plugin installs a splash page across your site so the only way in is through the admin area. Once you’re logged in, you can view the site as if it is fully available on the internet. The beauty of this method is that you don’t need to remember to unblock search engines when the site goes live.