A crawler engine based on Scrapy. I'm writing this to design a MVP (minimum viable product) for my upcoming venture, HayAuto.
This a trial, and will not be supported after some time. I will be writing a fully custom engine using Scala + Akka when I feel the time is right.
The goals of the Scala implementation are:
- Easy to configure: the options for this are either YAML/JSON or a custom DSL
- Easy to scale
- Distributable
- Highly fault tolerant
- Easy to extend
Anyways, enjoy!