To create a Docker image of a fully integrated server with a user interface UI to manage the scrapping and storage of up-to-date financial informations available on the internet.

A python web application to manage the extraction and storage of all Financial data available on the web. The software is written mainly in Python and uses popular librairies to create a web interface that master a scrapper engine which store the informations. It scraps from a broad list of sources (data providers, stock exchanges, etc.) in the web, gathers them to be stored in a PostgreSQL database. The manager shows a live visualisation of the state of the PostgreSQL database.

This project is a mix various Python projects found on Github. We link them to build a web interface that automate the mecanics of importing all captured data related to financial systems available on the web.

Visualise the "Big Picture" by looking at the live aggregation of all financial values stored in the database. Check is all the channels are opened and their informations up to date. Manage the sources

• Operating System: Ubuntu
• Docker:
• Language: Python3.9
• UserInterface: Django3.2
• Webscraper Library: Scrapy
• database: PostgreSQL

Json file

JavaScrip (TypeScrip) Django 3.2

Python 3.9 PostgreSQL Redis container (kubernetes) Tor

See more options, explanations and Pipenv usage in INSTALL.md.

• Establish a full list of financial assets available

1. Check if the source is available and up-to-date
2. Scrap the raw data
3. Clean the data and integrate them in the database

• a web interface
• serializers
• get Newsletters

To be implemented.

• https://github.com/je-suis-tm/web-scraping
• https://github.com/bmoscon/cryptofeed
• https://github.com/JerBouma/FinanceDatabase
• https://github.com/ckz8780/market-toolkit
• https://github.com/kootenpv/yagmail
• https://github.com/mcdallas/wallstreet
• https://github.com/JerBouma/FundamentalAnalysis
• https://github.com/theOGognf/finagg

Do not hesitate, leave a like, a feedback, a star, a follow, some money or even a contribution! This work needs contributors, every help will be welcomed. :)

For any questions left unaswered, please feel free to contact us:

1. at http://www.hadrian-advisors.com/
2. directly through GitHub @MaxWayne
3. or by sending us an email at: maximilien.pelletier@gmail.com