// h2o-3 dependency information
def h2oBranch = 'master'
def h2oBuildNumber = 'nnnn'
def h2oProjectVersion = "x.y.z.${h2oBuildNumber}"

repositories {
  // h2o-3 dependencies
  maven {
    url "https://s3.amazonaws.com/h2o-release/h2o-3/${h2oBranch}/${h2oBuildNumber}/maven/repo/"
  }
}

dependencies {
  compile "ai.h2o:h2o-core:${h2oProjectVersion}"
  compile "ai.h2o:h2o-algos:${h2oProjectVersion}"
  compile "ai.h2o:h2o-web:${h2oProjectVersion}"
  compile "ai.h2o:h2o-app:${h2oProjectVersion}"
}

# Build H2O
git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew build -x test

# Start H2O
java -jar build/h2o.jar

# Point browser to http://localhost:54321

git clone https://github.com/h2oai/h2o-3.git
cd h2o-3
./gradlew syncSmalldata
./gradlew syncRPackages
./gradlew build

#### Recipe 3:  Pull, clean, build, and run tests

#### Notes

 - We recommend using `./gradlew clean` after each `git pull`.

- Skip tests by adding `-x test` at the end the gradle build command line.  Tests typically run for 7-10 minutes on a Macbook Pro laptop with 4 CPUs (8 hyperthreads) and 16 GB of RAM.

- Syncing smalldata is not required after each pull, but if tests fail due to missing data files, then try `./gradlew syncSmalldata` as the first troubleshooting step.  Syncing smalldata downloads data files from AWS S3 to the smalldata directory in your workspace.  The sync is incremental.  Do not check in these files.  The smalldata directory is in .gitignore.  If you do not run any tests, you do not need the smalldata directory.
- Running `./gradlew syncRPackages` is supported on Windows, OS X, and Linux, and is strongly recommended but not required. `./gradlew syncRPackages` ensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using `./gradlew syncRPackages`. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):

### 4.2. Setup on all Platforms

##### Install required Python packages (prepending with `sudo` if unsuccessful)

  pip install grip
  pip install tabulate
  pip install wheel
  pip install scikit-learn

Python tests require:

  pip install scikit-learn
  pip install numpy
  pip install scipy
  pip install pandas
  pip install statsmodels
  pip install patsy

### 4.3. Setup on Windows

##### Step 1: Download and install [WinPython](https://winpython.github.io). 
From the command line, validate `python` is using the newly installed package by using `which python` (or `sudo which python`). [Update the Environment variable](https://github.com/winpython/winpython/wiki/Environment) with the WinPython path.

###### Step 2: Install required Python packages:

  pip install grip
  pip install tabulate
  pip install wheel

##### Step 3: Install JDK

Install [Java 1.7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html) and add the appropriate directory `C:\Program Files\Java\jdk1.7.0_65\bin` with java.exe to PATH in Environment Variables. To make sure the command prompt is detecting the correct Java version, run:

  javac -version

The CLASSPATH variable also needs to be set to the lib subfolder of the JDK: 

  CLASSPATH=/<path>/<to>/<jdk>/lib

##### Step 4. Install Node.js

Install [Node.js](http://nodejs.org/download/) and add the installed directory `C:\Program Files\nodejs`, which must include node.exe and npm.cmd to PATH if not already prepended. 

##### Step 5. Install R, the required packages, and Rtools:

To install these packages from within an R session, enter:

  R> install.packages("RCurl")
  R> install.packages("jsonlite")
  R> install.packages("statmod")
  R> install.packages(c("devtools", "roxygen2", "testthat"))

Install [R](http://www.r-project.org/) and add the preferred bin\i386 or bin\x64 directory to your PATH.

Note: Acceptable versions of R are >= 2.13 && <= 3.0.0 && >= 3.1.1.

To manually install packages, download the releases of the following R packages: 

- [bitops](http://cran.r-project.org/package=bitops)
- [devtools](http://cran.r-project.org/package=devtools)
- [digest](http://cran.r-project.org/package=digest)
- [Rcpp](http://cran.r-project.org/package=Rcpp)
- [RCurl](http://cran.r-project.org/package=RCurl)
- [jsonlite](http://cran.r-project.org/package=jsonlite)
- [roxygen2](http://cran.r-project.org/package=roxygen2)
- [statmod](http://cran.r-project.org/package=statmod)
- [stringr](http://cran.r-project.org/package=stringr)
- [testthat](http://cran.r-project.org/package=testthat).

cd Downloads
R CMD INSTALL bitops_x.x-x.zip
R CMD INSTALL RCurl_x.xx-x.x.zip
R CMD INSTALL jsonlite_x.x.xx.zip
R CMD INSTALL statmod_x.x.xx.zip
R CMD INSTALL Rcpp_x.xx.x.zip
R CMD INSTALL digest_x.x.x.zip
R CMD INSTALL testthat_x.x.x.zip
R CMD INSTALL stringr_x.x.x.zip
R CMD INSTALL roxygen2_x.x.x.zip
R CMD INSTALL devtools_x.x.x.zip

Finally, install [Rtools](http://cran.r-project.org/bin/windows/Rtools/), which is a collection of command line tools to facilitate R development on Windows.
>**NOTE**: During Rtools installation, do **not** install Cygwin.dll.

##### Step 6. Install [Cygwin](https://cygwin.com/setup-x86_64.exe)
**NOTE**: During installation of Cygwin, deselect the Python packages to avoid a conflict with the Python.org package. 

###### Step 6b. Validate Cygwin
If Cygwin is already installed, remove the Python packages or ensure that Native Python is before Cygwin in the PATH variable. 

##### Step 7. Update or validate the Windows PATH variable to include R, Java JDK, Cygwin. 

##### Step 8. Git Clone [h2o-3](https://github.com/h2oai/h2o-3.git)

If you don't already have a Git client, please install one.  The default one can be found here http://git-scm.com/downloads.  Make sure that command prompt support is enabled before the installation.

Download and update h2o-3 source codes:

    git clone https://github.com/h2oai/h2o-3

##### Step 9. Run the top-level gradle build:

    cd h2o-3
    ./gradlew.bat build

> If you encounter errors run again with `--stacktrace` for more instructions on missing dependencies.


### 4.4. Setup on OS X

If you don't have [Homebrew](http://brew.sh/), we recommend installing it.  It makes package management for OS X easy.

##### Step 1. Install JDK

Install [Java 1.7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html). To make sure the command prompt is detecting the correct Java version, run:

    javac -version

##### Step 2. Install Node.js:

Using Homebrew:

    brew install node

Otherwise, install from the [NodeJS website](http://nodejs.org/download/). 

##### Step 3. Install R and the required packages:

Install [R](http://www.r-project.org/) and add the bin directory to your PATH if not already included.

<a name="InstallRPackagesInUnix"></a>

Install the following R packages: 

- [RCurl](http://cran.r-project.org/package=RCurl)
- [jsonlite](http://cran.r-project.org/package=jsonlite)
- [statmod](http://cran.r-project.org/package=statmod)
- [devtools](http://cran.r-project.org/package=devtools)
- [roxygen2](http://cran.r-project.org/package=roxygen2) 
- [testthat](http://cran.r-project.org/package=testthat).

cd Downloads
R CMD INSTALL bitops_x.x-x.tgz
R CMD INSTALL RCurl_x.xx-x.x.tgz
R CMD INSTALL jsonlite_x.x.xx.tgz
R CMD INSTALL statmod_x.x.xx.tgz
R CMD INSTALL Rcpp_x.xx.x.tgz
R CMD INSTALL digest_x.x.x.tgz
R CMD INSTALL testthat_x.x.x.tgz
R CMD INSTALL stringr_x.x.x.tgz
R CMD INSTALL roxygen2_x.x.x.tgz
R CMD INSTALL devtools_x.x.x.tgz

To install these packages from within an R session:

    R> install.packages("RCurl")
    R> install.packages("jsonlite")
    R> install.packages("statmod")
    R> install.packages(c("devtools", "roxygen2", "testthat"))

##### Step 4. Git Clone [h2o-3](https://github.com/h2oai/h2o-3.git)

OS X should already have Git installed. To download and update h2o-3 source codes:

    git clone https://github.com/h2oai/h2o-3

##### Step 5. Run the top-level gradle build:

    cd h2o-3
    ./gradlew build

> If you encounter errors run again with `--stacktrace` for more instructions on missing dependencies.

### 4.5. Setup on Ubuntu 14.04

##### Step 1. Install Node.js

    curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
    sudo apt-get install -y nodejs

##### Step 2. Install JDK:

Install [Java 1.7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html). Installation instructions can be found here [JDK installation](http://askubuntu.com/questions/56104/how-can-i-install-sun-oracles-proprietary-java-jdk-6-7-8-or-jre). To make sure the command prompt is detecting the correct Java version, run:

    javac -version

##### Step 3. Install R and the required packages:

Installation instructions can be found here [R installation](http://cran.r-project.org).  Click “Download R for Linux”.  Click “ubuntu”.  Follow the given instructions.

To install the required packages, follow the [same instructions as for OS X above](#InstallRPackagesInUnix).

##### Step 4. Git Clone [h2o-3](https://github.com/h2oai/h2o-3.git)

If you don't already have a Git client:

    sudo apt-get install git

Download and update h2o-3 source codes:

    git clone https://github.com/h2oai/h2o-3

##### Step 5. Run the top-level gradle build:

    cd h2o-3
    ./gradlew build

> If you encounter errors, run again using `--stacktrace` for more instructions on missing dependencies.

> Make sure that you are not running as root, since `bower` will reject such a run.

### 4.6. Setup on Ubuntu 13.10

##### Step 1. Install Node.js

    curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
    sudo apt-get install -y nodejs
   

##### Steps 2-4. Follow steps 2-4 for Ubuntu 14.04

### 4.7. Setting up your preferred IDE environment

For users of Intellij's IDEA, generate project files with:

    ./gradlew idea

For users of Eclipse, generate project files with:

    ./gradlew eclipse


<a name="Launching"></a>
## 5. Launching H2O after Building

    java -jar build/h2o.jar


<a name="BuildingHadoop"></a>
## 6. Building H2O on Hadoop

Pre-built H2O-on-Hadoop zip files are available on the [download page](http://h2o.ai/download).  Each Hadoop distribution version has a separate zip file in h2o-3.

To build H2O with Hadoop support yourself, first install sphinx for python: `pip install sphinx`
Then start the build by entering  the following from the top-level h2o-3 directory:

    (export BUILD_HADOOP=1; ./gradlew build -x test)
    ./gradlew dist

This will create a directory called 'target' and generate zip files there.  Note that `BUILD_HADOOP` is the default behavior when the username is `jenkins` (refer to `settings.gradle`); otherwise you have to request it, as shown above.


### Adding support for a new version of Hadoop

In the `h2o-hadoop` directory, each Hadoop version has a build directory for the driver and an assembly directory for the fatjar.

You need to:

1.  Add a new driver directory and assembly directory (each with a `build.gradle` file) in `h2o-hadoop`
2.  Add these new projects to `h2o-3/settings.gradle`
3.  Add the new Hadoop version to `HADOOP_VERSIONS` in `make-dist.sh`
4.  Add the new Hadoop version to the list in `h2o-dist/buildinfo.json`

### Debugging HDFS

These are the required steps to debug HDFS in IDEA as a standalone H2O process.

Debugging H2O on Hadoop as a `hadoop jar` hadoop mapreduce job is a difficult thing to do. However, what you can do relatively easily is tweak the gradle settings for the project so that H2OApp has HDFS as a dependency.  Here are the steps:

1.  Make the following changes to gradle build files below
    *  Change the `hadoop-client` version in `h2o-persist-hdfs` to the desired version     
    *  Add `h2o-persist-hdfs` as a dependency to `h2o-app`
1.  Close IDEA
1.  `./gradlew cleanIdea`
1.  `./gradlew idea`
1.  Re-open IDEA
1.  Run or debug H2OApp, and you will now be able to read from HDFS inside the IDE debugger

`h2o-persist-hdfs` is normally only a dependency of the assembly modules, since those are not used by any downstream modules.  We want the final module to define its own version of HDFS if any is desired.

Note this example is for MapR 4, which requires the additional `org.json` dependency to work properly.

-----

<a name="Sparkling"></a>
## 7. Sparkling Water

Sparkling Water combines two open-source technologies: Apache Spark and H2O, our machine learning engine.  It makes H2O’s library of Advanced Algorithms, including Deep Learning, GLM, GBM, K-Means, and Distributed Random Forest, accessible from Spark workflows. Spark users can select the best features from either platform to meet their Machine Learning needs.  Users can combine Spark's RDD API and Spark MLLib with H2O’s machine learning algorithms, or use H2O independently of Spark for the model building process and post-process the results in Spark. 

**Sparkling Water Resources**:

* [Download page for pre-built packages](http://h2o.ai/download/) (Scroll down for Sparkling Water)
* [Sparkling Water GitHub repository](https://github.com/h2oai/sparkling-water)
* [README](https://github.com/h2oai/sparkling-water/blob/master/README.md)
* [Developer documentation](https://github.com/h2oai/sparkling-water/blob/master/DEVEL.md)

<a name="Documentation"></a>
## 8. Documentation

### Generate REST API documentation 

To generate the REST API documentation, use the following commands: 

    cd ~/h2o-3
    cd py
    python ./generate_rest_api_docs.py  # to generate Markdown only
    python ./generate_rest_api_docs.py --generate_html  --github_user GITHUB_USER --github_password GITHUB_PASSWORD # to generate Markdown and HTML

The default location for the generated documentation is `build/docs/REST`. 

If the build fails, try `gradlew clean`, then `git clean -f`. 

### Bleeding edge build documentation

Documentation for each bleeding edge nightly build is available on the [nightly build page](http://s3.amazonaws.com/h2o-release/h2o-3/master/latest.html).

-----

<a name="Community"></a>
## 9. Community

We will breathe & sustain a vibrant community with the focus of taking a software engineering approach to data science and empowering everyone interested in data to be able to hack data using math and algorithms.
Join us on google groups at [h2ostream](https://groups.google.com/forum/#!forum/h2ostream) and feel free to file issues directly on our [JIRA](http://jira.h2o.ai). 

Team & Committers

<a name="Advisors"></a>
## Advisors

Scientific Advisory Council

Systems, Data, FileSystems and Hadoop

<a name="Investors"></a>
## Investors