Robots Refresher: robots.txt — a flexible way to control how machines explore your website   developers.google.com

In this edition of the robots refresher series, Martin Splitt and John Mueller take a closer look at robots.txt as a flexible way to tell robots what you want them to do (or not do) on your website.