Python scripts for the daily tasks in particle physics (Run 2) - Daily Python Scripts (aka DPS)

Quick installation recipe:

# get the code from the repository
git clone https://github.com/BristolTopGroup/DailyPythonScripts
cd DailyPythonScripts

# for Run 1 code, please checkout the run1 branch
# git checkout run1

# get submodules:
git submodule init && git submodule update

# setup run:
source setup_conda.sh

# setup environment (using virtualenv for python):
source environment_conda.sh

# make sure matplotlib is up to date (should return 1.3.1 or above):
python -c 'import matplotlib; print matplotlib.__version__'

If working on soolin (or anywhere where dependencies like ROOT/latex/etc are not available), run it within CMSSW:

# install CMSSW and setup environment:
export SCRAM_ARCH=slc6_amd64_gcc491
scram p -n CMSSW_7_4_7_DPS_ CMSSW_7_4_7
cd CMSSW_7_4_7_DPS/src/
cmsenv
# This version comes with ROOT 6.02/05

# then install DailyPythonScripts according to the recipe above

ROOT ≥ 6.02

freetype

matplotlib ≥ 1.3

All plots/histograms provided by this package are based on either toy MC or simulated events from the CMS experiment. Real data is not included at any point.

config/* - files to save presets for available modules

data/* - for input/output of ROOT & JSON files (will be created by some scripts)

examples/* - generic examples for available modules

external/* - external projects (virtualenv, RooUnfold, etc)

plots/* - default output folder for plots (will be created by some scripts)

src/* - specific use of available modules

test/* - unit tests for available modules

tools/* - available modules

• Run experimental/BLTUnfold/merge_unfolding_jobs.sh to merge BLT unfolding CRAB output files (each sample needs to be merged into one file, cannot be split over several files)
• Move merged files to e.g.: /hdfs/TopQuarkGroup/mc/7TeV or 8TeV/v11/NoSkimUnfolding/BLT/

• run produceUnfoldingJobs.py with finebinning option turned on, on central sample only (run locally on soolin): python produceUnfoldingJobs.py -c=7 -f --sample=central
• Move fine binned unfolding file to /hdfs/TopQuarkGroup/results/histogramfiles/AN-14-071_6th_draft/7TeV or 8TeV/unfolding/
• Run the src/cross_section_measurement/00_pick_bins.py script to find new binning.
• Modify config/variable_binning (and TTbar_plus_X_analyser.cpp in AnalysisSoftware) with new binning

• Run python experimental/BLTUnfold/runJobsCrab.py with the last few lines commented out and uncommenting the line print len(jobs) to print the number of jobs.
• Update queue in experimental/BLTUnfold/submitBLTUnfold.description with the outputted number of jobs
• cd up to the folder containing DailyPythonScripts and tar --exclude='external/vpython' --exclude='any other large/unnecessary folders in DailyPythonScripts' -cf dps.tar DailyPythonScripts (tar file should be <= 100MB)
• Run experimental/BLTUnfold/produceUnfoldingHistogram.py script on merged files using HTCondor: condor_submit submitBLTUnfold.description to convert unfolding files to our binning. Check progress using condor_q your_username
• Once all jobs have finished, untar output files: tar -xf *.tar
• Output root files should be in a folder called unfolding. Move these new files to /hdfs/TopQuarkGroup/results/histogramfiles/AN-14-071_6th_draft/7TeV or 8TeV/unfolding/

• After running the Analysis Software, move the output files to /hdfs/TopQuarkGroup/results/histogramfiles/AN-14-071_7th_draft/7TeV or 8TeV using ```python experimental/move_BAT_output_files_to
• Find out how many merging jobs are needed using python experimental/merge_samples_onDICE.py -c 7(or 8) -n 1 --listJobs
• Modify the following lines in experimental/submitMerge.description: centre of mass energy: arguments = $(process) 7 or arguments = $(process) 8 number of jobs: enter the number of merging jobs for the centre of mass energy in question here e.g. queue 65
• cd up to the folder containing DailyPythonScripts and tar --exclude='external/vpython' --exclude='any other large/unnecessary folders in DailyPythonScripts' -cf dps.tar DailyPythonScripts (tar file should be <= 100MB)
• Merge the required BAT output files (e.g. SingleTop, QCD etc.) using condor_submit DailyPythonScripts/experimental/submitMerge.description

x_01_all_vars
x_02_all_vars
x_03_all_vars
x_04_all_vars
x_05_all_vars
x_98_all_vars
x_99_QCD_cross_checks
x_make_binning_plots
x_make_control_plots
x_make_fit_variable_plots

(script AN-14-071 runs all of these scripts automatically if you are confident everything will run smoothly(!))