PDA

View Full Version : What is the limit of a robot.txt file?


ninadordev
02-16-2018, 05:27 AM
Hello friends,

What is the limit of a robot.txt file?
Dam Ponting
02-16-2018, 06:28 AM
The /robots.txt file can also help to eliminate duplicate content issues that arise with blogging software, such as WordPress.There is limit of 500KB of the robot.txt file.
sravani24
02-16-2018, 08:55 AM
There is no limitations for obot.txt file but if you have a heavy robots.txt file, and it is beyond 500KB, then GoogleBot can get confused. If GoogleBot gets confused with your robots.txt it can cause serious issues with your website. So better to your website will be below 500KB.
RH-Calvin
02-23-2018, 05:13 PM
Google has a limit of only being able to process up to 500KB of your robots.txt file
sravani24
02-25-2018, 08:09 PM
Google has a limit of only being able to process up to 500KB of your robots.txt file

I think as per recent news we can upload up to 50 Mbps.