Robots.txt

A robots.txt file tells search engines which pages, directories, or file types in your site to avoid scanning. You really only need a robots.txt file if your site contains stuff you don't want search engines to index, or if you want to block specific search engines.

Your robots.txt file should live in the root or top-level directory for your website's files. Because your robots.txt is a public file, take care not to include any private or proprietary information in it.

How Search Engines Use It

Code Example

For demonstration purposes, only. Do not use.