Blocking Search Engine Indexing

TODO: review these notes

Note: Before go-live of a website, it could be a good idea to block the website from search engine indexing. This will prevent the website from being listed in search results, and prematurely sending traffic to the website. When the website is ready for launching the blocking needs to be removed.

Web Module

Website robots.txt page creation

• Expand Web Module, then click on website title where the robots.txt file is needed.
• The pagetree should open on the right side.
• At the top of the pagetree, click on the "Add page" button.
• Name the page: robots.txt.
• Do not add any preset to this page!
• Open the Page Settings for editing:
• General tab:
  • Title: robots.txt
  • Slug: robots.txt (the slug needs to be modified to exactly "robots.txt" and not robots-txt!)
  • Click on modify button, modify text, click on check to accept changes to the slug
  • Hide in menus: set checkmark
  • Hide in Sitemap: set checkmark
  • Click on "Update" button
• Advanced tab:
  • Navigate to the "Text" section of the page
  • Output text: set checkmark
  • Text Content:

User-agent: *
Disallow: /

Website launching

When the website is ready to go-live, simply disable the robots.txt page (or delete) and the effects of the robots.txt file on search engine indexing will be removed (as the webpage is disabled/deleted).