Results 1 to 5 of 5

Thread: What is Robots.txt?

  1. #1

    Default What is Robots.txt?

    Hello Friends,

    What is Robots.txt?

  2. #2
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    874

    Default

    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop Access
    Cheap Dedicated Servers | Linux Dedicated Server Hosting

  3. #3
    Join Date
    Aug 2017
    Location
    Pune
    Posts
    195

    Default

    The robots avoidance standard, otherwise called the robots prohibition convention or just robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots. The standard indicates how to educate the web robot about which zones of the site ought not be prepared or examined.

  4. #4

    Default

    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

  5. #5
    Join Date
    Feb 2016
    Location
    CA
    Posts
    176

    Default

    Robots.txt file help to control your website indexing from search engines.

Similar Threads

  1. What is Robots.txt File?
    By stuartspindlow2 in forum Web Design Solutions
    Replies: 15
    Last Post: 09-29-2017, 07:10 AM
  2. Why we use Robots.txt File?
    By riprook7 in forum Business Tools
    Replies: 3
    Last Post: 03-21-2017, 07:33 AM
  3. What is the use of robots.txt?
    By artijain1900 in forum Web Design Solutions
    Replies: 5
    Last Post: 10-06-2016, 09:35 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •