http://tribbloid.github.io/spookystuff/
... is a scalable query engine for web scraping/data mashup/acceptance QA. The goal is to allow the Web being queried and ETL'ed like a relational database.
SpookyStuff is the fastest big data collection engine in history, with a speed record of querying 330404 dynamic pages per hour on 300 cores.
- Apache Spark
- Selenium
- JSoup
- Apache Tika
- (build) Apache Maven
- (browser integration) PhantomJS/GhostDriver
- (drone integration) MAVLink
- Current implementation is influenced by Spark SQL and Mahout Sparkbinding.
Copyright © 2014 by Peng Cheng @tribbloid, Sandeep Singh @techaddict, Terry Lin @ithinkicancode, Long Yao @l2yao and contributors.
Published under ASF License, see LICENSE.