This project is read-only.
Project Description
SiteChecker is a tool to verify the presence and characteristics of a robots.txt file for a user-specified list of websites.

ScreenHunter_01 Jul. 19 11.59.jpg

SiteChecker was developed by Mando Group Labs before being released to CodePlex as an open source project.

Note: This tool has a number of known issues, but the basics are all in place. I'll be working on it in the coming month to get it to a baseline release.

Last edited Aug 5, 2012 at 8:20 PM by mattsalmon, version 10