Robots.txt


Robots.txt is a structural element on a website that tells search engines to ignore pieces of content or provides other instructions on how a website should be viewed by a search engine's bots or crawlers.