EasyChoose: A Continuous Feature Extraction and Review Highlighting Scheme on Hadoop YARN

this project is implementation of the iee paper -- https://ieeexplore.ieee.org/document/8432347

install hadoop and spark2.3.3 and set the spark master as Yarn 

run dfs and yarn of hadoop

download stanford parser and extract it and set the paths in first.py file for  stanford-parser.jar , stnford-parser-models.jar and englishPCFG.ser.gz  

upload reviews to the root folder of hdfs( command : "hdfs dfs -put /reviews /")
then 
#go to the /code directory and run these commands one by one

spark-submit --deploy-mode client --master yarn --class org.apache.spark.examples.SparkPi first.py

spark-submit --deploy-mode client --master yarn --class org.apache.spark.examples.SparkPi second.py

spark-submit --deploy-mode client --master yarn --class org.apache.spark.examples.SparkPi analysis.py

intermediate output folder --> /output (hdfs)
final output folder --> /fo  (hdfs)