Why we use Robots.txt File?

[riprook7]

Why we use Robots.txt File?

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[ArkPresentation]

Robots.txt is a file to allow or disallowed search engine to keep some pages privacy..

[sandrew272]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.