Ghost/core/server/public/robots.txt

User-agent: *
Sitemap: {{blog-url}}/sitemap.xml
Disallow: /ghost/
Disallow: /p/
Introduce default robots.txt 2014-01-28 17:05:59 +04:00			`User-agent: *`
Adds complete url for Sitemap in robots.txt closes #4589 - Adds {{blog-url}} to robots.txt - Replaces {{blog-url}} with blog.url from config 2014-12-13 14:47:35 +03:00			`Sitemap: {{blog-url}}/sitemap.xml`
Generate sitemap files Closes #623 - Add basic init and eventing scaffold - Add sitemap-index.xml generation - Broke out generators to individual files, added request handler - Add page, author and tag xml files; add index mapping - Add SiteMapManager unit tests - Add Generators tests - Cache invalidation headers for sitemap-*.xml - Redirect sitemap.xml to index and rename to sitemap-index - Handle page convert and publish/draft changes - Add very basic functional test for route existence - Add cache headers to sitemap routes 2014-10-28 03:41:18 +03:00			`Disallow: /ghost/`
🐛Disallowed indexing of preview pages (#9762) closes #9749 - disallow indexing of /p/ in robots.txt - add meta robots tag to preview pages - robots.txt wont' be applied for blogs not mounted to the root (i.e. https://example.com/blog/) 2018-09-17 12:29:47 +03:00			`Disallow: /p/`