Hello friends,
What is the limit of a robot.txt file?
Printable View
Hello friends,
What is the limit of a robot.txt file?
Google has a limit of only being able to process up to 500KB of your robots.txt file.
Googlebot reads the first 500 KB present in the robots.txt file.
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler.