DataWake prefetch scours the internet for user provided keywords, scrapes those pages and provides the user a ranked list of the website containing those keywords.
DataWake Prefetch is multi-tier software system consisting of client side applications (Firefox add-on), web servers (tangelo), and distributed backend platforms (kafka, storm, zookeeper, etc).
DataWake Prefetch uses a high-speed distributed crawler to find the relevant pages to your search.
- Git
- Docker
- Docker-Compose
- Working on Windows or OSX? Boot2Docker
Checkout DataWake Prefetch:
$ git clone https://github.com/sotera/datawake-prefetch.gitBuild the database:
$ cd dev-env/
$ docker-compose up -d mysql
$ ./init-db.shStart the DataWake Prefetch containers
$ docker-compose Build
$ docker-compose up -dNow check that the Docker containers are running.
$ docker-compose ps
Name Command State Ports
----------------------------------------------------------------------------------------
devenv_datawake_1 tangelo -c /usr/local/shar ... Up 0.0.0.0:80->80/tcp
devenv_dwsearch_1 scala -classpath build/lib ... Up
devenv_dwstream_1 /memex-datawake-stream/dev ... Up
devenv_dwurlcontents_1 scala -classpath build/lib ... Up
devenv_dwurlrank_1 scala -classpath build/lib ... Up
devenv_mysql_1 /entrypoint.sh mysqld Up 0.0.0.0:3336->3306/tcpRun the Firefox add-on:
$ cd ../FirefoxSidebar
$ cfx run