Detailed Setup Guide
(Optional) Configure specific pages to restrict crawling using robots.txt
(Optional) Configure restricted crawling across all pages using robots.txt
(Optional) Restrict search engine indexing across the entire shop
1. (Optional) Configure specific pages to restrict crawling using robots.txt
To restrict only certain pages from being crawled by search engine bots,
Note: Keep in mind that if a robots.txt entry is made for a product after it has been published, it could already have been crawled and indexed, and therefore will still appear in search engine results. To ensure previously crawled pages never appear in search engine results, consider setting up no indexing across the entire shop, detailed in section 3.
2. (Optional) Configure restricted crawling across all pages using robots.txt
First bullet should provide navigation instructions i.e. Catalog >Attributes >Product attributes
Sub-bullet for step 1
Sub-bullet for step a
Note: Keep in mind that if a robots.txt entry is made for a product after it has been published, it could already have been crawled and indexed, and therefore will still appear in search engine results. To ensure previously crawled pages never appear in search engine results, consider setting up no indexing across the entire shop, detailed in section 3.
3. (Optional) Restrict search engine indexing across the entire shop
First bullet should provide navigation instructions i.e. Catalog >Attributes >Product attributes
Sub-bullet for step 1
Sub-bullet for step a