format_converter

Convert file from csv to parquet through Apache Spark

Usage

export SAPRK_MASTER=spark://xxxx (deafult: local[1])
python convert.py --in s3://bucket/folder/from.csv --out s3://bucket/folder/to.parquet --mode [append/error/overwrite]

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
converter		converter
tests		tests
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
convert.py		convert.py
requirements.txt		requirements.txt