rss-scraper-django

A RSS scraper app which saves RSS feeds to a database and lets a user view and manage feeds they've added to the system.

Getting Started

Clone this repo:

git clone git@github.com:vishket/rss-scraper-django.git

And change into the main directory

cd rss-scraper-django

This app assumes you have docker & docker compose installed on your machine.

The app comprises of 5 (+1 nginx in prod) services, each of which can be spun up in individual docker containers.

To spin up the containers

docker-compose up -d --build

The website can be accessed at http://127.0.0.1:8000

To bring down containers

docker-compose down -v

Refer the production docker compose yaml file like so:

docker-compose -f docker-compose.prod.yml up -d --build

The website can be accessed at http://127.0.0.1:1337

docker-compose run celery pytest [--cov]

Some configuration to be aware of:

Feeds are scheduled to update every 30 mins
Max retry attempts for failed feed updates is 2 - with an exponential back-off set to 5 secs
Requests have a 10 sec timeout

These can all be changed in the settings file

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
apps		apps
nginx		nginx
rss_scraper		rss_scraper
.env.dev		.env.dev
.env.prod		.env.prod
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.prod		Dockerfile.prod
README.md		README.md
conftest.py		conftest.py
docker-compose.prod.yml		docker-compose.prod.yml
docker-compose.yml		docker-compose.yml
manage.py		manage.py
pytest.ini		pytest.ini
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt