GitHub - jennielili/ARL_Spark_Locality: A Spark version of ARL by exploring data locality

This is a Spark version of ARL, aiming to improve the performance by exploring data locality.

At first, we modify ical graph to put vis and mod_vis together, by this way, to reduce a join operation. ./run is the execution command.

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
ICAL_IO_Optimization		ICAL_IO_Optimization
arl		arl
develop_csv		develop_csv
examples		examples
LOWBD2.csv		LOWBD2.csv
README.md		README.md
SKA1_LOW_beam.fits		SKA1_LOW_beam.fits
arl.zip		arl.zip
cp		cp
full_python		full_python
imaging-pipelines-sip2.py		imaging-pipelines-sip2.py
pipeline-_spark_locality.py		pipeline-_spark_locality.py
pipeline-lqh.py		pipeline-lqh.py
pipeline-partitioning_auto_spark_empty.py		pipeline-partitioning_auto_spark_empty.py
pipeline-partitioning_locality.py		pipeline-partitioning_locality.py
requirements.txt		requirements.txt
run		run
sc128		sc128
sc16		sc16
sc256		sc256
sc32		sc32
sc512		sc512
sc64		sc64
spark-arl.conf		spark-arl.conf
test_plus_spark.xlsx		test_plus_spark.xlsx

jennielili/ARL_Spark_Locality