Log in

View Full Version : What is the limit of a robot.txt file?



shoppingswag
07-10-2019, 04:31 AM
What is the limit of a robot.txt file?

sinelogixtech
07-10-2019, 04:42 AM
Hi friends,
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

MVMInfotech18
07-10-2019, 10:35 AM
Your robots.txt file must be smaller than 500KB.

BangkokDinning
07-11-2019, 10:58 AM
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out.

RH-Calvin
07-12-2019, 06:52 AM
A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.