Skip to content

homich77/msa

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

* Need to install PostgreSQL

Ubuntu:
sudo -u postgres psql
Windows:
http://www.enterprisedb.com/products-services-training/pgdownload


* To work with Python need for Windows:
.Net Framework 3.5(include 2.0 and 3.0)
Microsoft Visual C++ Compiler Package for Python 2.7 (http://www.microsoft.com/en-us/download/details.aspx?id=44266)


* In PostgreSQL enter commands:
CREATE USER extr_admin WITH PASSWORD '123';
CREATE DATABASE extract_data;
GRANT ALL PRIVILEGES ON DATABASE extract_data to extr_admin;


* Create virtualenv and activate
virtualenv venv
Ubuntu:
. venv/bin/activate
Windows:
venv\Scripts\activate


* Install required modules
pip install -r requirements.txt


* Filling DB:
./manage.py migrate


* Create indexes in PostgreSQL:
create index on main_proxy(address);


* To run work
export DJANGO_SETTINGS_MODULE=extract_data.settings
python check_proxy.py


* To run craigslist spider
scrapy crawl msa

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published