To prevent search engines from crawling shopping pages and indexing them so that they are available via a Google search, the newest version of nopCommerce has several out-of-the-box features that can be leveraged to both prevent crawling of your shop (robots.txt) and to prevent indexing (no index) of the shop if a site has previously been crawled and indexed and therefore is available via Google search unless a noindex rule is added.
This feature is supported for: Robots.txt, the capability to prevent search engine crawling on some of all shop pages is available in version 3.0 and higher. Adding no index rules to the entire shop is available in all versions of ecommerce.
This feature is NOT supported for: Robots.txt, the capability to prevent search engine crawling on some of all shop pages is not available in any version prior to 3.0.
Key Terms
Robots.txt
Definition from Google
The robots
metatag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. Place the robotsmetatag in the<head>section of a given page…The robots
metatag instructs search engines not to show the page in search results. The value of thenameattribute (robots) specifies that the rule applies to all crawlers.
No Index Rules
Definition from Google
noindexis a rule set with either a<meta>tag or HTTP response header and is used to prevent indexing content by search engines that support thenoindexrule, such as Google. When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.
Feature Description
Many resorts use hidden categories and products that they do not want publicly available to guests that don’t have exclusive access. Additionally, many resorts want to first route customers through the marketing site in their research before directing them to shopping pages, for these reasons, it can be helpful to block the shop from search engine crawling and indexing.
Robots.txt essentially allows resorts to setup rules that prevent search engines from newly crawling pages, however, if the page has already been crawled and indexed it can still show up in search engine results. If this is not desirable, a no index rule can be added to the entire shop to prevent all shop pages from showing up in search results regardless of whether it was previously crawled and indexed or not. If a product or category is being newly created and it should not show up in search results, you can add the page to the set of no index rules in robots.txt and it will never appear in search results if done before publishing. Keeping up on robots.txt rules can allow you to keep many pages of your shop indexable and searchable, but will ensure newly created products and categories that should not be crawled do not appear in results.
For more detailed information on the differences between search engine crawling and indexing, refer to Google's documentation.
Note: Organic search accounts for about 1.8% of traffic across all Aspenware sites, so making your shop completely unavailable to search engines could restric traffic and influence metrics.