A crawler engine based on Scrapy. I'm writing this to design a MVP (minimum viable product) for my upcoming venture, HayAuto.

This a trial, and will not be supported after some time. I will be writing a fully custom engine using Scala + Akka when I feel the time is right.

The goals of the Scala implementation are:

• Easy to configure: the options for this are either YAML/JSON or a custom DSL
• Easy to scale
• Distributable
• Highly fault tolerant
• Easy to extend

Anyways, enjoy!