readme.txt
explain.txt
config.ini								program parameters' configuraton
crawler.py								main program
	
core.engine.py							load config file and manage the parser queue and download queue
core.downloader.py						downloader implementation assgins download tasks to thread pool
core.parser.py							parser implementation assgins parse tasks to thread pool
core.searchgoogle.py					Google search API implementation

models.configuration.py					load all configurations from local file and remote mysql
models.html.py							the data structure maintain the crawled page infomation
models.safe_dic.py						implementation of dictionary with lock
models.safe_queue.py					implementation of queue with lock
models.safe_loop_array.py				implementation of array with lock
models.status.py						system global variables

include.database_manager.py				interact with remote mysql
include.database.py						sql executer
include.log.py							implementation of logger
include.setting.py						read program parameters from local configuration file
include.thread_pool.py					implementation of a thread pool		

strategies.bookmarkhandler.py			handle page anchor
strategies.cgihandler.py				block url address with cgi in it
strategies.earlyvisithandler.py			block pages visited before
strategies.filetypehandler.py			decide whether a page is crawable according to its MIME type
strategies.linksextractor.py			extract links from a downloaded page
strategies.nestlevelhandler.py			block pages exceed a cetain depth in a site
strategies.omitindex.py					omit the part of 'index.htm','main.htm',etc with in a url
strategies.robotexclusionrulesparser.py	a robot exclusion rules parser 
strategies.robothandler.py				decide whether a page is crawable according to the robot.txt
strategies.schemehandler.py				scheme whitelist
strategies.urlextender.py				extend partial url

www.

Parser.Threadnum						The number of thread for parse
Parser.Nestlevel						The maximum depth of a page in a website

seed.keywords							The search key words
seed.result_num							The result num returned from the Google API

Mysql.host								mysql hostname
Mysql.user								mysql username
Mysql.passwd							mysql passwprd
Mysql.db								mysql database name

# Engine 
----------------------
	Engine is the main controller in the crawler.
	
		It has:
		-a downloader object and a parser object, 
		-two safe queues each for the download and parse tasks, 
		-a status object holding the global status variables for the crawler, 
		-a mysql manager, 
		-objects for every filter strategy,the safe dictionary within the object of earlyvisithandler actually maintains the tree structure of the visited url
		
	
	Once the Engine starts up:		
		-it first apply the filter rules to the seeds from the Google API, and then load the valid ones into the download queue;
		-it starts two threads each keep checking the download queue and parse queue, once a html task is found in the queues, it is then assigned to the downloader or the parser; 
		-The downloader will return html tasks to be parse and push them into the parse queue, and the parser will return the html tasks to be download and push them into the download queue;
		-it starts a thread keep checking the status of the crawler and post runtime info to remote mysql;
	

# Parser
----------------------
	Parser will assgin the parsing tasks passed from engine to the thread pool object it maintains
		
	Except applying filtering rules when loading seeds from Google API to download queue in the engine, we mainly apply all the filtering rules in parser:
		-robothandler 				check against robot exclusion rules through robot.txt, it maintains a dictionary,
									of which the key is the url's homesite, the value is a object of robotexclutionrulesparser.
		-earlyvisithandler			check against the url visited before, it maintains a dictionary,
									of which the key is the md5 hashcode of the url, the value is the url's corresponding html object.
		-cgihandler					block the url with cgi in it
		-bookmarkhandler			block the link of page anchor
		-filetypehandler			block the url according to the MIME type
		-nestlevelhandler			block url exceed a cetain depth in a site
		-omitindex					omit the part of 'index.htm','main.htm',etc with in a url
		-schemehandler				block the scheme outside the scheme whitelist
		-urlextender				return the complete url

# Downloader
----------------------
	Downloader will assgin the download tasks passed from engine to the thread pool object it maintains;
	
	The timeout is set as 2 secs, then it returns a exception;
	
	It saves download files to local directory

# Html
----------------------
	It stores the information of a certain url and its corresponding page data:
		_url 						initial url and the extended url
		_scheme						scheme of the url
		_hostname					hostname of the url
		_md5						md5 hash code of the url
		_id							download sequence
		_depth						distance to the initial seed
		_parent						parent Html object
		_return_code				200, 404, etc
		_data						text within the page
		_data_size					size of data
		_crawled_time				download time