GitHub - hijoe320/RSSNewsBot: A rss news bot to regularly check rss feeds and parse updated news articles

hijoe320 / RSSNewsBot Public

Notifications You must be signed in to change notification settings
Fork 0
Star 0

A rss news bot to regularly check rss feeds and parse updated news articles

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 34 Commits
rssnewsbot		rssnewsbot
testfiles		testfiles
.gitignore		.gitignore
DJI.txt		DJI.txt
FTSE.txt		FTSE.txt
LICENSE		LICENSE
NDX.txt		NDX.txt
README.txt		README.txt
feed_updater.py		feed_updater.py
load_feeds.py		load_feeds.py
scrapy.cfg		scrapy.cfg

Repository files navigation

Prerequests:

1. Setup and run redis
2. Setup and run mongodb

Star scraping

1. intiallise:
    python load_feeds.py --mongodb-uri mongodb://localhost:27017 -f DJI.txt --drop-existing -v

2. run rssspider:

3. run articlespider:


Other operations:
1. check scraping status
2. run another crawler on a different machine