PDA

View Full Version : Why we use Robots.txt File?


riprook7
02-15-2017, 07:13 AM
Why we use Robots.txt File?
RH-Calvin
02-15-2017, 08:38 AM
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
ArkPresentation
03-16-2017, 08:06 AM
Robots.txt is a file to allow or disallowed search engine to keep some pages privacy..
sandrew272
03-21-2017, 07:33 AM
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.