forked from cheekybastard/finance
-
Notifications
You must be signed in to change notification settings - Fork 0
mhumpher/finance-1
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
scrapy application for scraping some financial data REQUIRES: ========= scrapy (http://scrapy.org) w3lib (https://github.com/scrapy/w3lib) python-mysql INSTALLATION: ============= If you want to store scraped data: 1. Create mysql database and grant all privileges on a given user (recommended to create new one for this application) example: >> create database finance; >> grant all privileges on finance.* to 'finance_user'@'localhost' identified by "financepassword"; (replace finance, finance_user and financepassword by your custom strings) 2. Create database tables by dumping file finance.scheme: $ mysql -u root -p finance < finance.scheme 3. Create a file local_settings in same place as file settings.py, with mysql settings. In this case its content would be: MYSQLDB_NAME = "finance" MYSQLDB_USER = "finance_user" MYSQLDB_PASS = "financepassword" USAGE: ====== see scrapy --help PROJECT COMMANDS: ================= crawl - crawls data (use option --db for storing in configured database) all - spawns a crawl process for every spider (also admits --db option) list - lists all spiders report - prints comparative text reports (scrapy report --help for options)
About
scrapy tool for financial data scraping
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published
Languages
- Python 99.8%
- Shell 0.2%