scrapy spider and postgresql pipline for economist

diligent spider for EcoArchive

article summary is generated by nltk, you'll have to install nltk package and its corpora data: brown, averaged_perceptron_tagger and punkt first

use postgresql as storage, assuing that you have a postgresql service running on localhost and default port. set your own（db, uername, password）in setting.py

use crontab tools to add crawl job each week

1. install postgresql first

yum install postgresql-server postgresql-contrib

postgresql-setup initdb

2. install python packages

pip install -r requirement.txt

3. install nlktk corporas

ipython

import nltk

nltk.download('punkt')

nltk.download('brown')

nltk.download('averaged_perceptron_tagger')

mv collector/setting.py.example collector/setting.py

change the db user and password

scrapy crawl eco