a automatic python crawler
This is my graduation project. It's developing!
2015-2-19 v0.1.0: finish the core code.
2015-2-20 v0.1.1: fix bugs, recode the 'crawler.xml', more stable
2015-2-21 v0.1.2: fix bugs
bug:
-
some same result
-
some times list out of range in job.py[line:177]
-
unicode '\u200d' end of url, make crawler program error
-
saving snapshots can't working in some web sites