These are the primary software versions used in production, and you should be able to use them locally:

• Python 3.4.3
• Node 10
• Elasticsearch 5.4 (or later?)
• Java VM 1.8
• Ubuntu 14.04

• Install Xcode from the website or Mac App store because the local build will use some of Xcode's compilation tools.
• Install the Xcode commandline tools (the commandline tools may need to be reinstalled after OS and Xcode updates)
  • xcode-select --install

⭐ Note: You will need to open Xcode to accept the end-user agreement from the application, or from the commandline run:_

• sudo xcodebuild -license accept

• Verify that Homebrew is installed and working properly:
  • brew doctor

brew install libevent libmagic libxml2 libxslt openssl graphviz nginx
brew install freetype libjpeg libtiff littlecms webp # chromedriver temporarily broken
brew tap petere/postgresql
brew install postgresql@9.3
brew link --force postgresql@9.3
brew install --force node@10
brew link node@10 --force
brew cask install java8
brew install elasticsearch@5.6
pip3 install typing

⭐ Note: Elasticsearch 1.7 does not work with Java 9 :star: Note: To unlink elasticsearch 1.7 and install elasticsearch 5.6

• brew search elasticsearch
• brew install elasticsearch@5.6
• brew unlink elasticsearch@1.7 && brew link elasticsearch@5.6 --force

⭐ Note: Brew cannot find java8

• brew tap caskroom/versions # lookup more versions
• brew cask search java # java8 now in list
• brew cask install java8

⭐ Note: This additional step is required for new macOS Sierra installations

• brew cask install Caskroom/cask/xquartz

⭐ Note: Node version mangement with nvm: If you need to easily switch between node versions you may wish to use nvm instead (not required for most users)

• npm install -g nvm
• nvm install 10
• nvm use 10

⭐ Note: Node version mangement with homebrew: Switching node versions with homebrew (which is not package manager but not a typical version manager) is possible but is not as flexbible for this purpose, e.g.:

• brew install node@7
• brew unlink node@6 && brew link --overwrite --force node@7
• node --version
• brew unlink node@7 && brew link --overwrite --force node@6
• node --version

⚠️ Note: The current version of chromedriver doesn’t allow our BDD tests to pass, so you must install the earlier 2.33 version.

Visit https://chromedriver.storage.googleapis.com/index.html?path=2.33/ and download chromedriver_mac64.zip to your Downloads folder. Then move the file to /usr/local/bin. Verify you have the correct version installed by then entering:

• chromedriver --version

It should show, among other things, that you’re running chromedriver 2.33.

⚠️ Note: If you need to update Python dependencies (do not do this randomly as you may lose important brew versions of packages you need):

• rm -rf encoded/eggs (then re-run buildout below) and possibly
• brew update
• brew upgrade

Encoded requires a UNIX based system (Mac or Linux) and Python 3.4.3 (but works with 3.5.x):

• For local development on a Mac, follow the steps below. For Linux use apt-get or yum as your Linux flavor demands. You can consult cloud-config.yml for other steps.

• Note: Production is currently using the versions above thus your primary development should always work on that version, and you should test that your code works on versions that will be used in production.

• Linux: apt-get install python3.4-dev or equivalent

Mac OSX Python install instructions

The Python version management tool pyenv is very useful.

⚠️ Note: If you have previously installed python3 from homebrew, you may possibly wish to uninstall it (not required):

• brew uninstall --force python3

Install pyenv and set the default versions:

brew install pyenv
pyenv install 3.4.3
pyenv install 2.7.13
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo 'eval "pyenv shell 2.7.13 3.4.3"' >> ~/.bash_profile
source ~/.bash_profile

⭐ Note: Migrating pyenv Python packages

If you have previously installed a Python version from pyenv, and want to quickly migrate all your pypi packages to a new version (Python 2 to 2, and Python 3 to 3 only):

• brew install pyenv-pip-migrate

Example if you previously installed 2.7 which really is 2.7.0:

• pyenv install 2.7.13
• pyenv migrate 2.7 2.7.13

⭐ _Note: pyenv install fails with "ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?" for MAC OS High Sierra

Uninstall and re-install openssl using the following command when you install pyenv

• brew uninstall --ignore-dependencies openssl && brew install openssl && CFLAGS="-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew --prefix openssl)/lib" pyenv install <VERSION>

Set the correct Python for the current directory:

pyenv local 3.4.3

• pip3 install -U zc.buildout setuptools
• pyenv rehash
• buildout bootstrap
• bin/buildout

⭐ Note: If you have issues with postgres or the python interface to it (psycogpg2) you probably need to install postgresql via homebrew (as above)

⭐ Note: If you have issues with Pillow you may need to install new xcode command line tools: Update or Install Xcode from the Mac AppStore (reboot may be required) and re-run:

• xcode-select --install

⭐ Note: Clean ALL the Things! If you wish to completely rebuild the application or cleanly reload dependencies (:warning: long re-build time!):

• make clean && buildout bootstrap && bin/buildout

• Terminal window 1:
  In one terminal window startup the database servers and nginx proxy with:

  • bin/dev-servers development.ini --app-name app --clear --init --load

  This will first clear any existing data in /tmp/encoded. Then postgres and elasticsearch servers will be initiated within /tmp/encoded. An nginx proxy running on port 8000 will be started. The servers are started, and finally the test set will be loaded.

• Terminal window 2:
  In a second terminal, run the app with:

  • bin/pserve development.ini

Indexing will then proceed in a background thread similar to the production setup.

⭐ Note: If you run into a java stack trace when you run the app, it is worth checking if /usr/local/etc/elasticsearch/elasticsearch.yml might have the line: ‘path.plugins: /usr/local/var/lib/elasticsearch/plugins’. If it does, it needs to be commented out.

• Browse to the interface at http://localhost:8000/.

• To run specific tests locally:

  bin/test -k test_name

• To run with a debugger:

  bin/test --pdb

• Specific tests to run locally for schema changes:

  bin/test -k test_load_workbook
bin/test -k test_load_schema

• Run the Pyramid tests with:

  bin/test -m "not bdd"

• Run the Browser tests with:

  bin/test -m bdd -v --splinter-webdriver chrome

• Run the Javascript tests with:

  npm test

• Or if you need to supply command line arguments:

  ./node_modules/.bin/jest

• Test ALL the things!

  bin/test -v -v --splinter-webdriver chrome && npm test

Our Javascript is written using ES6 and JSX, so needs to be compiled using babel and webpack. Our CSS is written in the SCSS variant of Sass and also needs compilation using webpack.

• To re-build production-ready bundles, do:

  npm run build

  (This is also done as part of running buildout.)

• To build development bundles and continue updating them as you edit source files, run:

  npm run dev

The development bundles are not minified, to speed up building. The above command runs continually in your terminal window and watches for changes in Javascript and SCSS files, rebuilding the bundles as you make changes.

• After buildout you (if you have the correct permissions) can run for a single-node "cluster":

  bin/deploy

• The script above will spin up a server in the AWS cloud with the current branch, and with a computed nameserver alias based on the branch and your username. Note that this retrieves a Postgres database from the current backup, so "as is" applies specifically to the ENCODE Project (if you have forked the repo you will not have permission to retrieve the db). There are options to use a different branch and/or different instance name and also if you want to use AWS spot instances...and you can specify which AWS profile you want to use.

• Deploy script help (how to specify name, instance size, etc):

  bin/deploy --help

For all options, including setting up ES clusters (needed for full production). After indexing (currently 8+hrs) the machine can be downsized at AWS to an m4.2xlarge, unless you are planning to submit significant data to it.

To set up linting with Sublime Text 3 or Visual Studio Code, first install the linters:

pip3 install flake8
npm install -g eslint
npm install -g eslint-plugin-react

⚠️ Note: You don't have to use Sublime Text 3 but you must insure that linting in your editor behaves as it does in Sublime Text 3.

After first setting up Package Control (follow install and usage instructions on site), use it to install the following linting packages in Sublime Text 3:

• sublimelinter
• sublimelinter-flake8
• SublimeLinter-contrib-eslint (Sublime linter eslint instructions)
• babel (Babel instructions)

Sublime Linting with pyenv
To get Sublime to lint Python code using pyenv you must add the python version and paths and python_paths to your Sublime Linter Preferences. In Sublime Text, navigate to the user linter preferences:

• Sublime Preferences -> Package Settings -> Sublime Linter -> Settings-User

• Add the following (modify existing preference settings file or add this entire JSON object below if the file is blank):

{
    "user": {
        "@python": 3.4,
        
        "paths": {
            "linux": [],
            "osx": [
                "/Users/YOURUSERNAME/.pyenv/versions/3.4.3/bin/",
                "/Users/YOURUSERNAME/.pyenv/versions/2.7/bin/"
            ],
            "windows": []
        },
        "python_paths": {
            "linux": [],
            "osx": [
                "/Users/YOURUSERNAME/.pyenv/versions/3.4.3/bin/python3",
                "/Users/YOURUSERNAME/.pyenv/versions/2.7/bin/python"
            ],
            "windows": []
        }
    }
}

• Restart Sublime

Go to the Visual Studio Code marketplace and install these extensions:

• ESLint
• Python
• Sass

Versions

• python3 --version returns Python 3.4.3 (or variant like 3.4.x)
• node --version returns v10.15.0 (or variant like v6.x.y)
• elasticsearch -v returns Version: 1.7.6 (or variant like Version: 1.7.x)
• postgres --version returns postgres (PostgreSQL) 9.3 (or variant like 9.3.x)

Linting check

• Python: Open Sublime, make a change to a Python file, make an intentional syntax error (no : at the end an if evaluation). Warnings and errors should show on the left side of the line.

• JavaScript: Make a syntax error, or style error, in a JS file. Warnings and errors should show on the left side of the line.

bin/deploy --set-region-index-to True --volume-size 500