essay of the crucible | college application fees | http://forums.thewebhostbiz.com/index.php/an-essay-on-respect/ | aids awareness essay What is the limit of a robot.txt file?
Results 1 to 4 of 4

Thread: What is the limit of a robot.txt file?

  1. #1

    Default What is the limit of a robot.txt file?

    Hello friends,

    What is the limit of a robot.txt file?

  2. #2

    Default

    Google has a limit of only being able to process up to 500KB of your robots.txt file.

  3. #3
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    1,187

    Default

    Googlebot reads the first 500 KB present in the robots.txt file.
    Cheap VPS Hosting | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop Access
    Cheap Dedicated Servers | Linux Dedicated Server Hosting

  4. #4
    Join Date
    Nov 2017
    Location
    Delhi, India
    Posts
    26

    Default

    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler.

Similar Threads

  1. Meta Title Character Limit?
    By acmastechnologi in forum Business Tools
    Replies: 5
    Last Post: 04-19-2018, 06:26 AM
  2. What is the character limit for AdWords ad?
    By ninadordev in forum Business Tools
    Replies: 2
    Last Post: 03-31-2018, 09:13 AM
  3. Character limit of h1 and h2 in SEO?
    By acmastechnologi in forum Business Tools
    Replies: 1
    Last Post: 03-26-2018, 06:01 AM
  4. What is the limit of a robot.txt file?
    By ninadordev in forum Business Tools
    Replies: 4
    Last Post: 02-25-2018, 08:09 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •