Nautilus connectors kit is a tool which aim is getting raw data from different sources and store them as-is into different destinations (GCS, BQ, local files, etc.).
- Adobe Analytics 1.4
- Adobe Analytics 2.0
- Amazon S3
- Facebook Marketing
- Google Ads
- Google Analytics
- Google Cloud Storage
- Google Campaign Manager
- Google Display & Video 360
- Google Search Ads 360
- Google Search Console
- Google Sheets
- Oracle
- MySQL
- Radarly
- SalesForce
- Twitter Ads
- Yandex Campaign
- Yandex Statistics
- Google BigQuery
- Google Cloud Storage
- Amazon S3
- Local File
- Console (Debug)
NCK is divided in three main components : Readers, Streams, and Writers.
- Readers role is to read data from distant sources and transform it into stream object
- Streams role is to be read as file or line by line. There are local objects usable by writers
- Writers role is to write stream into distant location
nck could be consumed through a docker image or can be installed as package and then be used as library or a binary.
- Build Docker image using
make build_base_image
- Run image to get help
docker run --rm nautilus-connector-kit:latest --help
First install dependencies with:
pip install -r requirements.txt
Then run:
python nck/entrypoint.py
- Exec cmd
make dist
(it generates a source distrib and a wheel in the created directory dist/)
It is advised to do the following in a virtual env
python3 -m venv testenv; source testenv/bin/activate
pip wheel --wheel-dir=wheels -r requirements.txt (that creates folder of wheels for packages in requierements)
pip install --no-index --find-links=./wheels dist/[nck-file-generated].whl
pip install -e .
python setup.py install
- Run cmd
nckrun --help
(which is equivalent to python nck/entrypoint.py)
from nck.readers.dbm_reader import DbmReader