A scrapy-based spider for automated search and web data extraction. It is be able to solve captcha codes through Optical character recognition (OCR) processes, create the required requests, control cookies and sessions to perform their job successfully. It allows you to choose memory or disk to save the data dump to work with this.

The requirements are in the requirements.txt and system-requirements.txt files. The first one contains the requirements that you can install with pip, the last one can be in either way.

You must patch the pytesser.py requirement file with my provided custom patch named pytesser.py.patch.

Programmed in Python under GNU/Linux.

Use of scrapy framework to build the crawler.
Use of XML, JSON and XPath to work with data inputs.
Use of parsers to get and parse some data.
Use of custom patched pytesser to do ocr tasks.
Use of context managing to make it easy some tasks.
Use of operating system libraries to save data.
Use of http methods to interact with destination.
Use of PIP to install something.
Use of diff and patch for patching.
Use of virtualenv to make easy the python development.

Licensed under GPLv3. Read LICENSE file.

--

Àngel Ollé Blázquez