| Table of Contents |
|---|
Detailed Setup Guide
...
1. (Optional) Configure specific pages to restrict crawling using robots.txt
First bullet should provide navigation instructions i.e. Catalog >Attributes >Product attributes
Sub-bullet for step 1. Every step and click should be detailed so that someone could follow with little prior experience.
Sub-bullet for step a. Include screenshots of the step and use skitch to mark up screenshots. It can be downloaded here: https://www.techspot.com/downloads/5705-skitch.htmlTo restrict only certain pages from being crawled by search engine bots,
| Info |
|---|
Note: Keep in mind that if a robots.txt entry is made for a product after it has been published, it could already have been crawled and indexed, and therefore will still appear in search engine results. To ensure previously crawled pages never appear in search engine results, consider setting up no indexing across the entire shop, detailed in section 3. |
2. (Optional) Configure restricted crawling across all pages using robots.txt
First bullet should provide navigation instructions i.e. Catalog >Attributes >Product attributes
Sub-bullet for step 1
Sub-bullet for step a
| Info |
|---|
Note: Keep in mind that if a robots.txt entry is made for a product after it has been published, it could already have been crawled and indexed, and therefore will still appear in search engine results. To ensure previously crawled pages never appear in search engine results, consider setting up no indexing across the entire shop, detailed in section 3. |
3. (Optional) Restrict search engine indexing across the entire shop
First bullet should provide navigation instructions i.e. Catalog >Attributes >Product attributes
Sub-bullet for step 1
Sub-bullet for step a
...