Note: This repo is a collection of samples of webscrawlers built using the python Scrapy framework.For more details about Scrapy... prerequisites scrapy : 1.4 + scrapy-proxies Resources https://github.com/g0v/twly_crawler https://github.com/aivarsk/scrapy-proxies https://github.com/scrapy-plugins/scrapy-deltafetch https://github.com/scrapy-plugins/scrapy-magicfields