John Mueller said that since the robots.txt file is cached by Google for about 24 hours, it does not make much sense to dynamically update your robots.txt file throughout the day to control controlling. Google won't necessarily see that you don't want Google to crawl a page at 7 am and then at 9 am you do want Google to crawl that page.
Trending now


Never miss any SEO updates, news, tools, jobs, events! Subscribe to the weekly SEOFOMO newsletter
Leave Your Comment