GitHub - varunkapoor/Search-Engine: Search engine to crawl facebook videos and docs

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
crawler		crawler
framework		framework
loaders		loaders
styles		styles
templates		templates
README		README
main.py		main.py
models.py		models.py
term_loader.py		term_loader.py

Repository files navigation

This repository contains a simple search engine code. It is designed to crawl a youtube playlist and index the title, description and captions of the videos in the playlist.

some salient features are:

* It is built with the Bottle web framework and designed to run on Google App Engine.

* The web crawler script udacity_crawler.py uses BeautifulSoup, an HTML parsing library.

* The Robot Exclusion Rules parser.