Sombrio ** is a scalable search engine for the dark web. The goal of the project is to build a simple, robust search engine from scratch that can search through the dark web content.
The last time we tested (December 2016), the dark web did not provide us enough meaningful results to serve a query. However, each component was working and tested separately. Hence, this project can be used as a foundation to add more interesting features and build a more complex product.
Components
- Crawler: Crawls the onion websites
- Index Builder: Builds index chunk
- Index Server: Stores replicated index chunk
- MGMT: Serves as single master, schedules jobs for other servers and executes heartbeats messages
- Front-End: Serves a website with Google-like interface
- Back-End: Express server with caching and aggregation logic
API
Each component has its own Dockerfile that is ready to be built and tested.