Python scripts for the daily tasks in particle physics (Run 2) - Daily Python Scripts (aka DPS)

If working on soolin (or anywhere where dependencies like ROOT/latex/etc are not available), run it within CMSSW:

git clone https://github.com/BristolTopGroup/DailyPythonScripts
cd DailyPythonScripts
source bin/env.sh
# make sure matplotlib is up to date (should return 1.5.1 or above):
python -c 'import matplotlib; print matplotlib.__version__'
# test ROOT and rootpy
root -l -q
time python -c "import ROOT; ROOT.TBrowser()"
time python -c 'import rootpy'
time python -c 'from ROOT import kTRUE; import rootpy'

Setting up conda environment on your own machine, please have a look under the Conda section.

ROOT ≥ 6.04

freetype

matplotlib ≥ 1.5

All plots/histograms provided by this package are based on either toy MC or simulated events from the CMS experiment. Real data is not included at any point.

config/* - files to save presets for available modules

data/* - for input/output of ROOT & JSON files (will be created by some scripts)

examples/* - generic examples for available modules

plots/* - default output folder for plots (will be created by some scripts)

src/* - specific use of available modules

test/* - unit tests for available modules

tools/* - available modules

• Run experimental/BLTUnfold/merge_unfolding_jobs.sh to merge BLT unfolding CRAB output files (each sample needs to be merged into one file, cannot be split over several files)
• Move merged files to e.g.: /hdfs/TopQuarkGroup/mc/7TeV or 8TeV/v11/NoSkimUnfolding/BLT/

• run produceUnfoldingJobs.py with finebinning option turned on, on central sample only (run locally on soolin): python produceUnfoldingJobs.py -c=7 -f --sample=central
• Move fine binned unfolding file to /hdfs/TopQuarkGroup/results/histogramfiles/AN-14-071_6th_draft/7TeV or 8TeV/unfolding/
• Run the src/cross_section_measurement/00_pick_bins.py script to find new binning.
• Modify config/variable_binning (and TTbar_plus_X_analyser.cpp in AnalysisSoftware) with new binning

• Run python experimental/BLTUnfold/runJobsCrab.py with the last few lines commented out and uncommenting the line print len(jobs) to print the number of jobs.
• Update queue in experimental/BLTUnfold/submitBLTUnfold.description with the outputted number of jobs
• cd up to the folder containing DailyPythonScripts and tar --exclude='external/vpython' --exclude='any other large/unnecessary folders in DailyPythonScripts' -cf dps.tar DailyPythonScripts (tar file should be <= 100MB)
• Run experimental/BLTUnfold/produceUnfoldingHistogram.py script on merged files using HTCondor: condor_submit submitBLTUnfold.description to convert unfolding files to our binning. Check progress using condor_q your_username
• Once all jobs have finished, untar output files: tar -xf *.tar
• Output root files should be in a folder called unfolding. Move these new files to /hdfs/TopQuarkGroup/results/histogramfiles/AN-14-071_6th_draft/7TeV or 8TeV/unfolding/

• After running the Analysis Software, move the output files to /hdfs/TopQuarkGroup/results/histogramfiles/AN-14-071_7th_draft/7TeV or 8TeV using ```python experimental/move_BAT_output_files_to
• Find out how many merging jobs are needed using python experimental/merge_samples_onDICE.py -c 7(or 8) -n 1 --listJobs
• Modify the following lines in experimental/submitMerge.description: centre of mass energy: arguments = $(process) 7 or arguments = $(process) 8 number of jobs: enter the number of merging jobs for the centre of mass energy in question here e.g. queue 65
• cd up to the folder containing DailyPythonScripts and tar --exclude='external/vpython' --exclude='any other large/unnecessary folders in DailyPythonScripts' -cf dps.tar DailyPythonScripts (tar file should be <= 100MB)
• Merge the required BAT output files (e.g. SingleTop, QCD etc.) using condor_submit DailyPythonScripts/experimental/submitMerge.description

x_01_all_vars
x_02_all_vars
x_03_all_vars
x_04_all_vars
x_05_all_vars
x_98_all_vars
x_99_QCD_cross_checks
x_make_binning_plots
x_make_control_plots
x_make_fit_variable_plots

(script AN-14-071 runs all of these scripts automatically if you are confident everything will run smoothly(!))

DailyPythonScripts relies on a (mini)conda environment to provide all necessary dependencies. This section describes how to set up this environment from scratch. As a first step, download and install conda (you can skip this step if you are using a shared conda install, e.g. on soolin):

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
export CONDAINSTALL=<path to miniconda install> # e.g. /software/miniconda
bash miniconda.sh -b -p $CONDAINSTALL # or a different location

Next let us create a new conda environment with some base packages:

export PATH="$CONDAINSTALL/bin:$PATH"
export ENV_NAME=dps
export CONDA_ENV_PATH=$CONDAINSTALL/envs/${ENV_NAME}
conda config --add channels http://conda.anaconda.org/NLeSC
conda config --set show_channel_urls yes
conda update -y conda
conda install -y psutil
conda create -y -n ${ENV_NAME} python=2.7 root=6 root-numpy numpy matplotlib nose sphinx pytables rootpy

Then activate the environment and install the remaining dependencies

source activate $ENV_NAME
# install dependencies that have no conda recipe
pip install -U uncertainties tabulate

At this point you should have all necessary dependencies which you can try out with:

root -l -q
time python -c "import ROOT; ROOT.TBrowser()"
time python -c 'import rootpy'
time python -c 'from ROOT import kTRUE; import rootpy'