democracy essays | writing a short report | essay help now What is Robots.txt?
Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: What is Robots.txt?

  1. #1

    Default What is Robots.txt?

    Hello Friends,

    What is Robots.txt?

  2. #2
    Join Date
    Jun 2013


    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS Hosting | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop Access
    Cheap Dedicated Servers | Linux Dedicated Server Hosting

  3. #3
    Join Date
    Aug 2017


    The robots avoidance standard, otherwise called the robots prohibition convention or just robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots. The standard indicates how to educate the web robot about which zones of the site ought not be prepared or examined.

  4. #4


    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

  5. #5
    Join Date
    Feb 2016


    Robots.txt file help to control your website indexing from search engines.

  6. #6
    Join Date
    Nov 2017

    Red face what is robots.txt file?

    robots.txt is a text file that lists web pages which are allowed and disallowed from search engine crawling.

  7. #7
    Join Date
    May 2016


    robots.txt is file which is control whole website through search engine crwaller

  8. #8

    Default What is inventory software?

    What is inventory software?

  9. #9


    Robots.txt file is a text file that tells search engines to index the pages which you want to search engines to crawl & which files you want to ignore.

  10. #10
    Join Date
    Apr 2018


    A robots.txt file consists of one or more rules. Each rule blocks (or or allows) access for a given crawler to a specified file path in that website.

    Here is a simple robots.txt file with two rules, explained below:

    # Rule 1
    User-agent: Googlebot
    Disallow: /nogooglebot/

    # Rule 2
    User-agent: *
    Allow: /

Similar Threads

  1. What is Robots.txt File?
    By stuartspindlow2 in forum Web Design Solutions
    Replies: 15
    Last Post: 09-29-2017, 07:10 AM
  2. Why we use Robots.txt File?
    By riprook7 in forum Business Tools
    Replies: 3
    Last Post: 03-21-2017, 07:33 AM
  3. What is the use of robots.txt?
    By artijain1900 in forum Web Design Solutions
    Replies: 5
    Last Post: 10-06-2016, 09:35 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts