Robot and LLM Crawler settings - Webfactory

We've added new Crawler File Settings, giving you greater control over the information your website publishes to search engines, web crawlers and AI services.

You can now manage both robots.txt and llms.txt directly from your Website Manager. This can be found by going to:

Website Editor > Settings > Search Optimisation > Crawler Files

Robots.txt

You can now add your own rules beneath Webfactory's standard robots.txt configuration. This allows you to:

• Request that compliant crawlers avoid specific areas of your website.
• Reduce crawling of duplicate, filtered or low-value pages.
• Provide different instructions for specific crawler services.

Your published file can be viewed at: yourwebsite.co.uk/robots.txt

Important: Only edit these rules if you understand how robots.txt works. Incorrect settings could prevent search engines from indexing parts of your website. Robots.txt is not a security feature and should not be used to protect private content.

LLMs.txt

You can also publish an optional llms.txt file containing a Markdown summary of your website, along with links to your most important pages. This can help participating AI services better understand your website and locate key content.

Your published file can be viewed at: yourwebsite.co.uk/llms.txt

Please note that support for llms.txt varies between AI services, so publishing one does not guarantee that your website will be referenced or used by AI tools.

Both robots.txt and llms.txt settings are completely optional and are only intended for websites with specific crawler requirements or those wishing to provide additional guidance to AI services.