In order to avoid unwanted content within the research indexes, webmasters can instruct spiders never to crawl sure files or directories through the common robots.txt file in the foundation directory on the domain. In addition, a web site may be explicitly excluded from a search engine's databases through the use of a meta tag certain to robots (or