RC

RC_Balance_Cash

 time python cli.py -nopp 2  -dcf config/db_config.DEV.json -pcf config/proc/RC/rest_iq/RC_Balance_Cash.json  --proc_params  \
 '2019/07/31' e***1

RC_Balance_Cash (auto detect month)

 time python cli.py -nopp 2  -dcf config/db_config.DEV.json -pcf config/proc/RC/rest_iq/RC_Balance_Cash.json  --proc_params  \
 'auto' e***1

IQ -> Vertica

ME_FinancingPosition

time python cli.py -nopp 3 -dcf config/db_config.DEV.json -pcf config/proc/data_sync/iq_file_vert/delta_load.json --proc_params   \
Position.ME_FinancingPosition "WHERE AccountingDate='2019-08-31' and client=223906"  test -ld 100

Vertica->IQ

ME_FinancingPosition

time python cli.py -nopp 3 -dcf config/db_config.DEV.json -pcf config/proc/data_sync/vert_file_iq/delta_load.json --proc_params   \
Position.ME_FinancingPosition "WHERE AccountingDate='2019-08-31' and client=223906"  test -ld 100

FICC IQ->IQ

DY_FiccDistribution

time python cli.py -nopp 3 --dump  -dcf config/db_config.DEV.json -pcf config/proc/ficc/iq_rest_iq/DY_FiccDistribution.json  --proc_params  \
Repo '2019-08-30' e5***@1

ME_FiccDistribution

time python cli.py -nopp 3 --dump  -dcf config/db_config.DEV.json -pcf config/proc/ficc/iq_rest_iq/ME_FiccDistribution.json \
 --proc_params  Repo '2019-08-31' e5***@1

GFIN

export G3_TEMP_DIR=/tmp/gfin
time ~/python3/bin/python3 cli.py -nopp 1 -dcf config/db_config.DEV.json -pcf config/proc/g3/dir_gfin_vertica/gfin.json --proc_params  \
./legacy/gfin

Params: --workingDir

GRDS

 time ~/python3/bin/python3 cli.py -nopp 1 -dcf config/db_config.DEV.json -pcf config/proc/g3/dir_grds_vertica/grds.json --proc_params  ./legacy/grds

Params: --workingDir

GTX

time python3 cli.py  -nopp 1 -dcf config/db_config.DEV.json -pcf config/proc/g3/dir_gtx_vertica/gtx.json --proc_params  ./legacy/gtx

Params: --workingDir

FICC

DY_FiccDistribution

 time python cli.py -nopp 3 --dump  -dcf config/db_config.DEV.json -pcf config/proc/ficc/iq_rest_iq/DY_FiccDistribution.json \
 --proc_params  Repo '2019-05-31' e5569eb7-e333-4e28-ad77-0f224a7d2499@1

ME_FiccDistribution

 time python cli.py -nopp 3 --dump  -dcf config/db_config.DEV.json -pcf config/proc/ficc/iq_rest_iq/ME_FiccDistribution.json \
 --proc_params  Repo '2019-05-31' e5569eb7-e333-4e28-ad77-0f224a7d2499@1

IQ proc -> Vertica

Direct

DY_Position_TD

time ~/python3/bin/python3 cli.py -nopp 27 -dcf config/db_config.DEV.json  -rte DEV -pcf config/proc/position/iq_proc_dir_v/DY_Position_TD.json   --proc_params \
223906 "EOD" 2019/8/1 "DESK" "CEFL" "DEFAULT" "REGULAR" "1" "0" "NONE" "*" "*" "N" "0" "0" "NONE" "NONE" "ALL" "0" "MONTH_END" "N" "ALL" "*" "0" "*" "N" "*"

Using automation

223906

time python tools/loader/position/iq_proc_dir_v.py -t DY_Position_TD -yr 2019 -mf 8 -mt 8 -df 1 -dt eom  -cl 223906  -cr ACCT  --dry

223907 (all of aug 2019)

time python tools/loader/position/iq_proc_dir_v.py -t DY_Position_TD -yr 2019 -mf 8 -mt 8 -df 1 -dt eom  -cl 223907  -cr ACCT  --dry

223907 ( aug 22 2019)

time python tools/loader/position/iq_proc_dir_v.py -t DY_Position_TD -yr 2019 -mf 8 -mt 8 -df 22 -dt 22  -cl 223907  -cr ACCT  --dry

IQ proc -> IQ

Direct

DY_Position_TD

time ~/python3/bin/python3 cli.py -nopp 27 --dump -dcf config/db_config.PROD.json  -rte PROD -pcf config/proc/position/iq_proc_dir_iq/DY_Position_TD.json -ld 37  --proc_params \
223906 "EOD" 2019/1/1 "DESK" "CEFL" "DEFAULT" "REGULAR" "1" "0" "NONE" "*" "*" "N" "0" "0" "NONE" "NONE" "ALL" "0" "MONTH_END" "N" "ALL" "*" "0" "*" "N" "*"

ME_Position_TD

time ~/python3/bin/python3 cli.py -nopp 27 --dump -dcf config/db_config.PROD.json  -rte PROD  -pcf config/proc/position/iq_proc_dir_iq/ME_Position_TD.json -ld 30  --proc_params 223906 "EOD" 2019/1/31 "DESK" "CEFL" "DEFAULT" "REGULAR" "1" "0" "NONE" "*" "*" "N" "0" "0" "NONE" "NONE" "ALL" "0" "FULL"      "N" "ALL" "*" "0" "*" "Y" "*"

Using automation

DY_Position_TD

time python tools/loader/position/iq_proc_dir_iq.py -t DY_Position_TD -yr 2019 -mf 1 -mt 1 -cl 223906 -df 1 -dt eom -cr DESK  -bu CEFL -ld 37

ME_Position_TD

time python tools/loader/position/iq_proc_dir_iq.py -t ME_Position_TD -yr 2019 -mf 1 -mt 1 -cl 223906  -cr DESK -bu CEFL  -ld 30

GFIN snap ->SQLite

 ~/python27/bin/python2 cli.py -nopp 1 -dcf config/db_config.DEV.json -pcf config/proc/g3/file_gfin_lite/gfin.json --proc_params  ./gfin

Open SQLite cli:

python ~/litecli/main.py dump/lite/20190829_160315/SQLite.db

describe table: PRAGMA table_info([Barrier]) Count: Select count(*) from Barrier

IQ->IQ (position)

Test all

Report over Outlook emails

Max Elapsed

Data Sync

From table name
From query
- Using file

Data Dump

Docs

Examples

Position
- IQ->SF
- SQL->SF
PNL
- IQ->SF
Ficc
- SF->REST->SF
File load (Subledger)
- File->SF
Data Sync (HydraPNLEntries)
- IQ->SQL
Data Dump (HydraPNLEntries)
- IQ->File
- SQL->File
Teardown
- IQ vs SQLServer (dmbalance)

Workflows

Stream
- IQ->StreamSlicer->S3->Snowflake
From File
- Dir->S3->Snowflake
Memory
- IQ->memory->SQLServer
REST
- Snowflake->REST->Snowflake
From IQ
- IQ->memory->SQLServer
- IQ->StreamSlicer->S3->Snowflake
From Snowflake
- Snowflake->REST->Snowflake
Into Snowflake
S3
- IQ->StreamSlicer->S3->Snowflake
- Dir->S3->Snowflake
From SQLServer
- SQL->StreamSlicer->S3->Snowflake
Into SQLServer
- IQ->memory->SQLServer
Into File
- Data dump
Compare 2 sides
- DB->vs<-DB

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
config		config
docs		docs
include		include
release/Snowflake		release/Snowflake
tools		tools
.gitignore		.gitignore
README.md		README.md
__init__.py		__init__.py
cli.py		cli.py
cli.spec		cli.spec
data.sh		data.sh
dataMigration.sh		dataMigration.sh
data_load.md		data_load.md
iqcli.sh		iqcli.sh
k.sh		k.sh
kinit.sh		kinit.sh
litecli.sh		litecli.sh
odbc.sh		odbc.sh
push.sh		push.sh
pyinst.sh		pyinst.sh
requirements.txt		requirements.txt
run.sh		run.sh
sk.sh		sk.sh
ss2.py		ss2.py
t.sh		t.sh
test.py		test.py
test.sh		test.sh
v.py		v.py
vcli.sh		vcli.sh

pie-crust/etl

Folders and files

Latest commit

History

Repository files navigation

RC

RC_Balance_Cash

RC_Balance_Cash (auto detect month)

IQ -> Vertica

ME_FinancingPosition

Vertica->IQ

ME_FinancingPosition

FICC IQ->IQ

DY_FiccDistribution

ME_FiccDistribution

GFIN

GRDS

GTX

FICC

DY_FiccDistribution

ME_FiccDistribution

IQ proc -> Vertica

Direct

DY_Position_TD

Using automation

IQ proc -> IQ

Direct

DY_Position_TD

ME_Position_TD

Using automation

DY_Position_TD

ME_Position_TD

GFIN snap ->SQLite

IQ->IQ (position)

Report over Outlook emails

Docs

Examples

Workflows

About

Topics

Resources

Stars

Watchers

Forks

Languages