This is an extension to the amazing scrapy-rotating-proxies library this with very minor changes. The main target of this library is to get proxies dynamically when the scraper is running. This library automatically fetches freely available lists of proxies from free-proxy-list.net.

pip install scrapy-rotating-free-proxies

After installing you need to do just add following two variables in settings.py of your Scrapy project :

ROTATING_PROXY_LIST_PATH = '/my/path/proxies.txt'

DOWNLOADER_MIDDLEWARES = {
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
}

For further details on using this library, refer to the original readme.

Thank you!