Overview: Block Search Engine Crawling and Indexing

Owned by Jenni Buonanni
Last updated: Nov 21, 2024 by Sarah Holst
2 min read

This feature is supported for: Robots.txt, the capability to prevent search engine crawling on some of all shop pages is available in version 3.0 and higher. Adding no index rules to the entire shop is available in all versions of ecommerce.

This feature is NOT supported for: Robots.txt, the capability to prevent search engine crawling on some of all shop pages is not available in any version prior to 3.0.

Key Terms

Robots.txt

Definition from Google

The robots meta tag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. Place the robots meta tag in the <head> section of a given page…

The robots meta tag instructs search engines not to show the page in search results. The value of the name attribute (robots) specifies that the rule applies to all crawlers.

No Index Rules

Definition from Google

noindex is a rule set with either a <meta> tag or HTTP response header and is used to prevent indexing content by search engines that support the noindex rule, such as Google. When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.

Feature Description

Resorts often have hidden categories and products exclusively for certain guests. To control access and guide customers through the marketing site before shopping, blocking the shop from search engine crawling and indexing is helpful.

Using robots.txt, resorts can set rules to prevent search engines from new crawling of pages. However, if a page has already been crawled and indexed, it may still appear in search results. To avoid this, a "no index" rule can be applied to the entire shop, preventing all shop pages from showing up in search results, regardless of prior crawling and indexing.

For newly created products or categories that shouldn't be in search results, adding them to the "no index" rules in robots.txt before publishing prevents them from appearing in search results. Keeping up with robots.txt rules allows you to keep most shop pages indexable and searchable while ensuring that newly created products and categories stay hidden from search engine results.

For more detailed information on the differences between search engine crawling and indexing, refer to Google's documentation.

NOTE: Organic search accounts for about 1.8% of traffic across all Aspenware sites, so making your shop completely unavailable to search engines could restric traffic and influence metrics.