Robots.txt

The robots.txt is a text file that resides in the root directory of your pubwriter site and provides search engine crawlers instructions as to which pages they can crawl and index, during the crawling and indexing process.

Robots.txt Tester

Typical

User-agent: *
Allow: /
Disallow: /buttons
Disallow: /admin
Disallow: /analytics
Sitemap: /sitemap.xml

Go Deeper