Robots.txt Parser Based on Google’s Open Source Parser realrobotstxt.com

It used to be possible to test specific robots.txt configurations in Search Console, but the tester did not match real googlebot behaviour, and eventually Google chose to deprecate that tool. This site’s own robots.txt file exposes the problem of using the open source checker without a wrapper. You can copy-and-paste it here to see that both googlebot and googlebot-image should be DISALLOWED from crawling /bar/. Without handling this kind of case specifically, the open source project will not get this right.