Sometimes it makes sense to block search engines from indexing your website.
For example, you might buy a brand new domain name and build your WordPress website on a live server to save the hassle of moving it once it’s finished (make sense for beginners), or you might want to redesign an existing site on a temporary subdomain.
In some cases, you might want to build a private website that you don’t want to be found through any search engine. You could be using it for a specific project, an intranet or for a private family blog.
If any of these situations apply to you, and you don’t want people to find your site through organic search results, there’s a setting inside WordPress you can use to instruct search engines like Google not to crawl or index your pages.
Sadly, this directive isn’t followed 100% of the time. So there’s still a chance search engines may index some piece of content or other from your site. It’s a small chance, but a chance nonetheless.
With that said, unless you password protect your site, you have little choice but to use the method I’m about to describe.
How to stop search engines indexing your WordPress website or blog
Log into the admin area of your WordPress website and navigate to Settings > Reading and tick/check the box next to Discourage search engines from indexing this site.
Checking this box does two things:
It adds this command to the source code of every page on your site.
<meta name="robots" content="noindex,follow">
These two commands – noindex and follow – tell search bots to not index the page but to follow the links within the page to discover more content. Because this code exists on every page, Google won’t show your pages in the search results.
And it adds this to the robots.txt file
User-agent: * Disallow: /
This tells search engine bots to disallow all content. When a bot lands on your domain it checks for a robots.txt file. If it finds one containing the protocols above, it won’t crawl and index your site.
At least, that’s the theory.
The problem with each of these methods is that you can’t guarantee your pages and other content won’t get indexed.
That’s all you have to do to block search engines indexing your content.
Working on your site while blocking search engines
If you want to work on your site and block people from viewing the pages while it’s in development, you need to install a maintenance mode plugin.
There are tons to choose from.
A maintenance mode plugins installs a splash page across your site so the only way in is through the admin area. Once you’re logged in, you can view the site as if it is fully available on the internet and work on getting your content right.
This is what the splash page from the Maintenance plugin looks like on a live site. All the text is editable so you can customise it for your site and situation.
You MUST remember to change the settings when your site is ready!
When your site is ready for the world to see, you really must go back to the settings page and untick/uncheck the Search Visibility button. If you don’t do that, the search engines won’t index your website.
At this stage you might be asking yourself how search engines find out about a new site.
There are a few ways, in the case of WordPress powered sites, it’s usually by pinging.
It’s built-in to WordPress. You find it in Settings > Writing.
At the bottom of the page is this section (note – it’s only visible if search engines are NOT blocked):
This is what you see if you’ve blocked search engines.
When you publish a new post, or update an old one, WordPress sends out a notification message to site called Ping-o-matic.com.
Ping-o-matic notifies its subscribers that there is new content on your site, or older content has changed, and they send their web crawling programs (spiders) to index or re-index your content. All this happens electronically and in a split-second.
When you’re ready for the search engines to index your site, head over to Ping-o-matic, enter your site’s details the way I have in this image, and hit the Send Pings button.
The web crawlers should visit your site straight away and start indexing your content.
It might take a while for Google to show it in the search results, but there’s a chance of it appearing within a few hours.