Small Crawler written in Python 2.7

Basic Features:

1. Uses Max Depth to stop crawling
2. Can crawl outside domain URL's
3. Checks for http response 200

Needs:

1. Multiprocessing support
2. Handle 301 and 302 url's
3. Add crawled data into a database
4. Respect robots.txt