This project was one of my projects for CS 5700. The goal of the project was to login to a website setup for the course called Fakebook. After this the crawler is supposed to crawl the links within the website to find 5 secret flags.
Our crawler is multithreaded and HTTP-Pipelined so it runs in a very efficent manner by using upto 300 threads on a machine. Part of the restrictions of the project was to implement the HTTP protocol on our own and not use any libraries related to HTTP.