- ipython_notebooks/
  This is ipython test code, that we were writing and sharing among each other
  while we were testing various features. Evaluator can ignore this.

- data/
We parsed data from two data source and pre-processed it to produce pickle files.
readable by python. As the size of actual pickle files is huge, we are turning in one
411M of decent size pkl file. You can use this file for testing.

- TaskTestData/
This code has been provided by the task organizers to test the performance of our model.
Use the following commands to execute scoring script:
> perl TaskTestData/key/score.pl <OUTPUT_FILE> ../TaskTestData/key/gold -t best
> perl TaskTestData/key/score.pl <OUTPUT_FILE> ../TaskTestData/key/gold -t oot

- scripts/
a) BigThesaurus.py
    - Used by our model to get candidate substitutions.
    - Teser can Ignore.
b) SpaceSubClassModel.py
    - We subclass Space class from the Dissect python toolkit to support
      advanced vector operations.
    - If using our model, one must have Dissect toolkit installed on their machines.
    http://clic.cimec.unitn.it/composes/toolkit/
    - Tester can ignore.
c) combine_sm_models.py
    - This script is used to combine co-occurance count files generated from
    different data sources.
    - Tester can ignore.
d) dissect_sample.py
    - This is a test script to run dissect model.
e) gemma_parser.py
    - This script is used to parse WikiCorpus ttp://www.lsi.upc.edu/~nlp/wikicorpus.
f) uwack_parser_pos.py
    - This script is used to parse huge http://wacky.sslmit.unibo.it/doku.php?id=corpora
    - Out of these parsing scirpts is co-occurance counts file, and two other files
      which Dissect expects as input.
g) upper_bound.py
    - Test Script
    - Tester can Ignore.
h) thesaurus_data
    - Actual thesaurus data, which we parsed from the BigHugeCorpus site.
    - Tester can Ignore.
i) thesaurusParser.py
    - This script was used to parse thesaurus data and store it in JSON format.
    - Tester can Ignore.
j) google_model_tester.py
    - We also ran some initial tests in google word2Vec. However, this is not used
    in our final approah.
    - Tester can Ignore.
k) get_pos_split_on_output.py
    - Test script.
    - Tester can ignore.
l) dissect_model_tester.py
    - This is the main script, which we run for running all kind of simulations.
    - Run python ./dissect_model_tester.py --help to get list of all options.
    - <NOTE>
        - Install Dissect tool kit before running this script.
        - You should have BigThesaurus.py and thesaurus_data in the same folder
          as this script.
    - This model can be passed in raw co-occurance counts, which can be used to 
      produce pickle files, which can be used later.
    - There are many features in this script. However, only we good combinations 
      provide us with good resutls.

   Sample invocations:

   python /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/scripts/dissect_model_tester.py --rows_file /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/data/150_row_file_pos --cols_file /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/data/150_col_file_pos --sm_file /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/data/150_matrix_file_pos --xml_input /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/TaskTestData/test/lexsub_test_cleaned.xml --top_features 5000 --save_location /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/data/150_gemma_plmi.pkl --ppmi --lwindow 1 --rwindow 1  --left_right_add  --thesaurus 1.0 --output_to gemma_plmi_lradd_1.0t_5000_test_no_pos

--rows_file, --cols_file and --sm_file are not required, if you have pkl file. This 
invocation would do the required computations and all also save the pickle file for 
futture use.

>> Equivalent invocation would be:

python /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/scripts/dissect_model_tester.py --pkl_file /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/data/150_with_ppmi.pkl --xml_input /Users/gnanda/nlp/final_project/English-Lexicalized-Text-Substituion/TaskTestData/test/lexsub_test_cleaned.xml --top_features 5000 --ppmi --lwindow 1 --rwindow 1  --left_right_add  --thesaurus 1.0 --output_to gemma_plmi_lradd_1.0t_5000_test_no_pos

# This would generate various output files, using thesaurus can candidate substitutions. Use 
 window size of 1 for creating context vectors. Selects top 5000 features from the vector
 space. Use PPMI (Positive Pointwise Mututal information) to select feature set.


m) graphical_model_tester.py
    - This is the main script used to call the Graphical Model based approach used to create the final output file. 
    It can be used to run all kind of simulations.
    - Run python ./graphical_model_tester.py --help to get list of all options.
    - <NOTE>
        - Install Dissect tool kit before running this script.
        - You should have BigThesaurus.py and thesaurus_data in the same folder
          as this script.
    - This model can be passed in raw co-occurance counts, which can be used to 
      produce pickle files, which can be used later.
    - There are many features in this script, which can be used to control the tarining data, the count og how many words on the left or the right need to be included n the graph being created.
    - In order to control 
   Sample invocations:
   python scripts/graphical_model_tester.py --pkl_file ./data/1_lemma_pos.pkl --xml_input ./TaskTestData/test/lexsub_test_cleaned.xml --lwindow 1 --rwindow 1  --left_right_add  --thesaurus 1.0 --output_to output_nanda_6 --start_word 1 --end_word 2100