Search engine base (crawler, indexer and parser) using Python, Celery, RabbitMQ, CouchDB and Whoosh.

Inspired by GoogleScraper's author who wrote in the project's README that his goal was to make the whole internet searchable and give ordinary people like him independent access to the internet's information again.

The goal of this project however is only to experiment with different methods of crawling and indexing websites.