| Table of Contents |
|---|
Detailed Setup Guide
(Optional) Configure specific pages to restrict crawling using robots.txt
(Optional) Configure restricted crawling across all pages using robots.txt
(Optional) Restrict search engine indexing across the entire shop
1. (Optional) Configure specific pages to restrict crawling using robots.txt
This is recommended to prevent search engines from crawling hidden categories, voucher product urls, and any page that should not be indexed by search engines.
To restrict only certain pages from being crawled by search engine bots, configure your robots.txt entry for your shop by going to Configuration > Settings > General Settings.
Locate the robots.txt section and expand it. It will be pre-loaded with some default restrictions setup by nopcommerce.
...
| Info |
|---|
Note: Keep in mind that if a robots.txt entry is made for a product after it has been published, it could already have been crawled and indexed, and therefore will still appear in search engine results. To ensure previously crawled pages never appear in search engine results, consider setting up no indexing across the entire shop, detailed in section 3. |
2. (Optional) Configure restricted crawling across all pages using robots.txt
To restrict only certain pages from being crawled by search engine bots, configure your robots.txt entry for your shop by going to Configuration > Settings > General Settings.
Locate the robots.txt section and expand it. It will be pre-loaded with some default restrictions setup by nopcommerce.
To disable crawling of the entire site enter
/*at the end of the localizable disallow path.
| Info |
|---|
Note: Keep in mind that if a robots.txt entry is made for a product after it has been published, it could already have been crawled and indexed, and therefore will still appear in search engine results. To ensure previously crawled pages never appear in search engine results, consider setting up no indexing across the entire shop, detailed in section 3. |
3. (Optional) Restrict search engine indexing across the entire shop
To restrict the entire shop from being indexed by search engines, go to Configuration > Settings > All Settings.
Search for the Setting: ‘seosettings.customheadtags’ and edit it.
Add the following value and save.
Code Block <meta name="robots" content="noindex">
If something is already added to this setting, as it is also used to define Adobe Analytics Credentials, keep what is added to the setting and include the value above on a new line.
...