What is the use of robots.txt?

[artijain1900]

Hello Guys,


I would like to know that What is the use of robots.txt? tell me ..

[quantumsound]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[seojesica]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.


Yes, Thanks, I completely agree with you.

[manchun]

Robots.txt is use for block website url from google bot.

[stuartspindlow]

A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want accessed by search engine crawlers.