GitHub - fone4u/scrapy-googledir-mysql: scrapy's googledir example with mysql store pipeline using twisted's adbapi

fone4u / scrapy-googledir-mysql Public

forked from rmax/dirbot-mysql

Notifications You must be signed in to change notification settings
Fork 0
Star 0

scrapy's googledir example with mysql store pipeline using twisted's adbapi

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
db		db
googledir		googledir
README		README
scrapy-ctl.py		scrapy-ctl.py

Repository files navigation

1. create database
    $ mysql -uroot
    mysql> create database googledir;
2. create table
    $ mysql -uroot googledir < db/mysql.sql
3. start crawling
    $ ./scrapy-ctl.py crawl directory.google.com

DB credentials are hardcoded in `googledir/pipelines.py`