How to Stop Search Engines from Crawling a WordPress Site

Do you want to stop search engines from crawling your WordPress website? There are many circumstances where you need to stop crawling for your website. So here in this article, we will guide you through the ways to stop the search engine from crawling and indexing your WordPress website.

Reason for stop crawling a WordPress website

There are many reasons a user can stop crawling and indexing their website. Many users don’t know about staging websites or how to work on a local development site, so they are developing their websites live. In this situation, they don’t want Google to find their website when it’s in development.

There are many organizations or companies that run their websites for their own documentation and project management. They don’t want to share their confidential documents publicly. So they need to stop crawling their website.

There are many users who run private websites and don’t want to show them on Google. So they need to stop indexing and crawling their website.

There are different methods to stop the search engine from crawling a WordPress website. Here we will discuss some easy ways for you.

Discourage search engines to index your WordPress site

This is the quickest and easiest method but it does not fully protect your website from being crawled. This is a built-in feature of WordPress that offers you restricted crawling and indexing of your website. Login to your WordPress admin dashboard and navigate to Settings > Reading page. Now go to the Search engine visibility option and check the box. This will discourage search engines from indexing this site.

After enabling this feature WordPress adds a no-index follow on your website header. WordPress also modifies your website’s robot.txt file so that it can ask search engines not to index your WordPress website. It only sent a request to a search engine not to crawl, but it’s up to the search engine to accept or regret the request. Most of the time search engines respect the request.

Password protect your website using cPanel

Using the Cpanel you can password-protect your entire website so that no one can access your website content. For this login to your Cpanel account dashboard and navigate to the Directory Privacy option.

Now find the public_html option and click on the Edit option on the right side.

If you have multiple websites installed then click on the public_html option then click on the Edit option of the website which you want to password protect.

Now you need to set up password protection for your website. From this page enable the Password protect this directory option. Then you can enter a name for your protected directory. After that click on the Save option.

Now WordPress will set access permission for your directory. Click on the Go Back option and it will go to the page where you can create a username and password for your website to access it.

Here on this page create a username and password for your website. You will need this username and password when you try to access your website.

Request search engine not to crawl individual page

In this method search engines can crawl your website but you can prevent any specific page or post from being crawled. For this, you need to install the Yoast SEO plugin on your website. Install and activate the plugin and then you can use this plugin feature to restrict search engines for any specific post or page.

Edit the post or page that you don’t want to index, and scroll down to find the Yoast SEO setting. You will find it at the bottom.  Click on the Advanced Tab where you will find Allow search engines to show this Post in search results. option. Using the dropdown many select No. 

There is also another option: Should search engines follow links on this Post? Select the No option here. This will restrict this specific page to index on search engines.

Wrapping Up

Following the process, you will be able to stop search engines from crawling your WordPress website. You can see our other articles to learn How To Fix Error Too Many Redirects Issue In WordPress

How To Fix 500 Internal Server Error In WordPress

How To Fix the “This Site Ahead Contains Harmful Programs” Error In WordPress