Random links from the web
- Python 2.7.x:
python --version - Install pip
- Download PhantomJS driver binary (regarding your os) and put it at the root of the project as
phantomjs - Install python packages
pip install virtualenv
virtualenv venv
source venv/bin/activate # . venv/bin/activate
pip install -r requirements.txt- Download unwanted urls list (used to filter links)
wget http://sbc.io/hosts/alternates/fakenews-gambling-porn/hosts > unwanted_urlsPut the configuration into config.json file with the following format
{
"PORT": 4000,
"APP_SECRET_KEY": "abcd1234",
"WEBSITE_TITLE": "MyWebsite",
"MONGO_URI": "mongodb://localhost:27017",
"feeder": {
"PHANTOM_JS_DRIVER_ARGS": ["--web-security=no", "--ssl-protocol=any", "--ignore-ssl-errors=yes"],
"PAGE_LOAD_TIMEOUT": 120,
"DEFAULT_USERNAME": "randomery",
"MOBILE_USER_AGENT": "Mozilla/5.0 (Linux; Android 7.1.2; Nexus 5X Build/N2G48C) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.98 Mobile Safari/537.36",
"DESKTOP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
}
}mkdir -pv mongodb/data # create data folder
mongod --dbpath mongodb/data # start the db
python server.py # start the servermkdir -pv mongodb/data # create data folder
mongod --dbpath mongodb/data # start the db
uwsgi -H venv --ini uwsgi.ini # start the servermkdir -pv /mongodb/data # create data folder
pip install supervisor
# TODOPut some rss links into rss_sources.json file with the following format
{
"sources": [
"https://example.com/rsslink",
"https://example2.com/othersslink"
]
}Run the feeder
python -c "from lib.feeder import insert_all_links;insert_all_links()"Process links added by users (:warning: infinite loop)
python -c "from lib.worker import job_loop;job_loop()"Spot some issues, bad typos or bad indentation
pip install pylint # install
pylint lib server.py # runSome of them can still run even after stopping the application, just kill them
kill -9 $(ps aux | grep phantomjs | awk '{print $2}')