Prerequisite: if you haven't, add this line to your ~/.profile:

source /cvmfs/cms.cern.ch/cmsset_default.sh

Skip this part if you already have conda running on uaf.

From within your home directory on the uaf, follow the below instructions to set up the tools to run coffea. We do this in a virtual environment, using the miniconda environment management package. You might get some error messages about packages that couldn't get uninstalled that you (usually) can ignore.

curl -O -L https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b 

Add conda to the end of ~/.bashrc, so relogin after executing this line

~/miniconda3/bin/conda init

Stop conda from activating the base environment on login

conda config --set auto_activate_base false
conda config --add channels conda-forge

Install package to tarball environments

conda install --name base conda-pack -y

Create environments with as much stuff from anaconda

conda create --name coffeadev uproot dask dask-jobqueue matplotlib pandas jupyter hdfs3 pyarrow fastparquet numba numexpr -y

And then install residual packages with pip

conda run --name coffeadev pip install jupyter-server-proxy coffea autopep8 jupyter_nbextensions_configurator klepto yahist

conda deactivate
conda create --name workerenv uproot dask dask-jobqueue pyarrow fastparquet numba numexpr -y
conda activate workerenv

Pack it

conda pack -n workerenv --arcroot workerenv -f --format tar.gz --compress-level 9 -j 8 --exclude "*.pyc" --exclude "*.js.map" --exclude "*.a"

I had to update my (local) python version, so you might have to run

conda install python=3.8.6

If you want to fix bugs, get the very latest version of coffea or are just adventurous you can install coffea direct from the github repository

git clone https://github.com/CoffeaTeam/coffea
cd coffea
pip install --editable .[dev]

Full instructions are given here.

Deactivate the conda environment with

conda deactivate

and then follow the below instructions. Some of the code lives within CMSSW_10_2_9. Ideally set it up in a fresh directory, recipe as follows:

cmsrel CMSSW_10_2_9
cd CMSSW_10_2_9/src
cmsenv
git cms-init

git clone --branch tW_scattering https://github.com/danbarto/nanoAOD-tools.git PhysicsTools/NanoAODTools

cd $CMSSW_BASE/src

git clone --recursive https://github.com/danbarto/tW_scattering.git

scram b -j 8
cmsenv

cd tW_scattering

You should be set up now. The following steps have to be repeated everytime you log in to uaf (from within tW_scattering)

source activate_conda.sh

To start a jupyter server just do

( conda activate coffeadev && jupyter notebook --no-browser )

In order to use jupyter you need to establish another ssh connection from your computer:

ssh -N -f -L localhost:8893:localhost:8893 johndoe@uaf-10.t2.ucsd.edu

Where 8893 represents the port (check the output of the jupyter server, uaf-10 should be replaced with the uaf you're working on, and johndoe by your user name, of course

When you ran the above setup you should have created a daskworkerenv.tar.gz file. Move this into tW_scattering/Tools/. If you lost the tarball, just rerun

conda pack -n daskworkerenv --arcroot daskworkerenv -f --format tar.gz \
    --compress-level 9 -j 8 --exclude "*.pyc" --exclude "*.js.map" --exclude "*.a"

Then, run packCode.sh, which is located in tW_scattering. This script downloads the latest version of the tW_scattering code and creates a tarball that's shipped to the DASK workers.

ipython -i start_cluster.py

Starts a cluster with 50 workers. The scheduler address is automatically dumped into a text file so that it can be picked up easily in any notebook using coffea. You can get the status of the cluster by just typing c into the ipython prompt.

To deactivate the environment, just type conda deactivate

Uninstall the jupyter kernel if you're having problems with it:

jupyter kernelspec uninstall coffeadev

and then reinstall it again

python -m ipykernel install --user --name=coffeadev
jupyter nbextension install --py widgetsnbextension --user
jupyter nbextension enable widgetsnbextension --user --py

If you already have a jupyter server running on the uaf, a different port than 8893 might be used. In this case, alter the ssh -N -f ... command so that it matches the ports. To stop a running jupyter server that is running but you can't find anymore, run ps aux | grep $USER. This will return you the list of processes attributed to your user. You should also find sth like

dspitzba 3964766  1.3  0.0  87556 44720 pts/17   S+   05:03   0:02 python /cvmfs/cms.cern.ch/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_2_9/external/slc6_amd64_gcc700/bin/jupyter-notebook --no-browser --port=8893

To stop this process, just type kill 3964766. In this case, 3964766 is the process id (PID) of the jupyter server process.

If a port is already used on your machine because of a not properly terminated ssh session, run the following command on your computer ps aux | grep ssh. This returns a similar list as before. There should be a job like

daniel           27709   0.0  0.0  4318008    604   ??  Ss    8:11AM   0:00.00 ssh -N -f -L localhost:8893:localhost:8893 uaf-10.t2.ucsd.edu

Similarly, you can stop the process by running kill 27709.

Latest recommendations at https://cms-analysis.github.io/HiggsAnalysis-CombinedLimit/#setting-up-the-environment-and-installation

cd $CMSSW_BASE/src
git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit
cd HiggsAnalysis/CombinedLimit
git fetch origin
git checkout v8.2.0
scramv1 b clean; scramv1 b # always make a clean build

cd $CMSSW_BASE/src
wget https://raw.githubusercontent.com/cms-analysis/CombineHarvester/master/CombineTools/scripts/sparse-checkout-https.sh; source sparse-checkout-https.sh
scram b -j 8

• v0p1p12: data processing (no MC submitted yet)
• v0p1p11: dilep/trilep skim only! added variables for ttH lepton ID
• v0p1p10: intermediate version
• v0p1p9: used for initial tW scattering studies presented in SnT