启信宝企业信息爬虫

使用REQUESTIUM实现requests和selenium进行差量化抓取

架构设计

REQUESTIUM的原因是需要登录和更新COOKIE,同时还需要解决验证码的问题(目前还未解决). 但是缺点是只能单进程运行一个REQUESTIUM进程,每个进程运行一个账号,所以需要多个账号多进程. 且由于对IP和账号都进行了限制,所以需要多个账号,高间断的请求.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.idea		.idea
.DS_Store		.DS_Store
README.md		README.md
category_mapping.json		category_mapping.json
chromedriver		chromedriver
concat_urls.txt		concat_urls.txt
configs.py		configs.py
driver.py		driver.py
geckodriver		geckodriver
geckodriver.log		geckodriver.log
json2mysql.py		json2mysql.py
jsonintomongo.py		jsonintomongo.py
parser.py		parser.py
parser.pyc		parser.pyc
process_urls.py		process_urls.py
qichacha.py		qichacha.py
qixin_driver.py		qixin_driver.py
qixin_driver.sublime-workspace		qixin_driver.sublime-workspace
qixin_parser.py		qixin_parser.py
qixin_parser.pyc		qixin_parser.pyc
qixin_rules.py		qixin_rules.py
qixin_rules.pyc		qixin_rules.pyc
qixinbaotj.py		qixinbaotj.py
requestium_driver.py		requestium_driver.py
result.json		result.json
result20180822-20180822.json		result20180822-20180822.json
rules.xslt		rules.xslt
shangbiaowang.py		shangbiaowang.py
tianyancha.py		tianyancha.py
tj.csv		tj.csv
tj.json		tj.json
urls.json		urls.json
urls.txt		urls.txt