A very simple web crawler. Built using Python 3.7.

Upon cloning this project it is possible to run the web crawler instantly using:

python3 src/webcrawler.py 'http://www.your-url.com'

webcrawler provides the following options which are not required:

• -l --limit: the maximum number of URLs to display
• -ua --user-agents: strings to use as User-Agent header in request