Hello Guys,
I would like to know that What is the use of robots.txt? tell me ..
Hello Guys,
I would like to know that What is the use of robots.txt? tell me ..
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
█ Cheap VPS Hosting | $1 VPS Hosting
█ Windows VPS Hosting | Windows with Remote Desktop Access
█ Cheap Dedicated Servers | Linux Dedicated Server Hosting
viaje de norte de la india | viaje de la india | tour india en espanol | viaje a la india desde mexico | viajes a la india | viaje a rajasthan india, Paquete de viaje india y nepal, agencia de viajes a la india, viaje a la india desde colombia, viaje a la india desde peru, viaje a la india desde chile, agencia viajes india
Robots.txt is use for block website url from google bot.
A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want accessed by search engine crawlers.