Welcome to H2O 3¶

Additional Requirements¶

• Hadoop: Hadoop is not required to run H2O unless you want to deploy H2O on a Hadoop cluster. Supported versions are listed on the Download page (when you select the Install on Hadoop tab) and include:
  • Cloudera CDH 5.4 or later
  • Hortonworks HDP 2.2 or later
  • MapR 4.0 or later
  • IBM Open Platform 4.2

Refer to the Hadoop Users section for detailed information.

• Conda 2.7 or 3.5 repo: Conda is not required to run H2O unless you want to run H2O on the Anaconda Cloud. Refer to the Anaconda Cloud Users section for more information.
• Spark: Version 1.6 or 2.0. Spark is only required if you want to run Sparkling Water.

New User Quick Start¶

New users can follow the steps below to quickly get up and running with H2O directly from the h2o-3 repository. These steps guide you through cloning the repository, starting H2O, and importing a dataset. Once you’re up and running, you’ll be better able to follow examples included within this user guide.

1. In a terminal window, create a folder for the H2O repository. The example below creates a folder called “repos” on the desktop.

user$ mkdir ~/Desktop/repos

2. Change directories to that new folder, and then clone the repository. Notice that the prompt changes when you change directories.

user$ cd ~/Desktop/repos
repos user$ git clone https://github.com/h2oai/h2o-3.git

3. After the repo is cloned, change directories to the h2o folder.

repos user$ cd h2o-3
h2o-3 user$

4. Run the following command to retrieve sample datasets. These datasets are used throughout this User Guide and within the Booklets.

h2o-3 user$ ./gradlew syncSmalldata

At this point, determine whether you want to complete this quick start in either R or Python, and run the corresponding commands below from either the R or Python tab.

# Download and install R:
# 1. Go to http://cran.r-project.org/mirrors.html.
# 2. Select your closest local mirror.
# 3. Select your operating system (Linux, OS X, or Windows).
# 4. Depending on your OS, download the appropriate file, along with any required packages.
# 5. When the download is complete, unzip the file and install.

# Start R
h2o-3 user$ r
...
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
>

# Copy and paste the following commands in R to download dependency packages.
> pkgs <- c("methods","statmod","stats","graphics","RCurl","jsonlite","tools","utils")
> for (pkg in pkgs) {if (! (pkg %in% rownames(installed.packages()))) { install.packages(pkg) }}

# Run the following command to load the H2O:
> library(h2o)

# Run the following command to initialize H2O on your local machine (single-node cluster) using all available CPUs.
> h2o.init()

# Import the Iris (with headers) dataset.
> path <- "smalldata/iris/iris_wheader.csv"
> iris <- h2o.importFile(path)

# View a summary of the imported dataset.
> print(iris)

  sepal_len    sepal_wid    petal_len    petal_wid        class
-----------  -----------  -----------  -----------  -----------
        5.1          3.5          1.4          0.2  Iris-setosa
        4.9          3            1.4          0.2  Iris-setosa
        4.7          3.2          1.3          0.2  Iris-setosa
        4.6          3.1          1.5          0.2  Iris-setosa
        5            3.6          1.4          0.2  Iris-setosa
        5.4          3.9          1.7          0.4  Iris-setosa
        4.6          3.4          1.4          0.3  Iris-setosa
        5            3.4          1.5          0.2  Iris-setosa
        4.4          2.9          1.4          0.2  Iris-setosa
        4.9          3.1          1.5          0.1  Iris-setosa
[150 rows x 5 columns]
>

# Before starting Python, run the following commands to install dependencies.
# Prepend these commands with `sudo` only if necessary.
h2o-3 user$ [sudo] pip install -U requests
h2o-3 user$ [sudo] pip install -U tabulate
h2o-3 user$ [sudo] pip install -U future
h2o-3 user$ [sudo] pip install -U six

# Start python
h2o-3 user$ python
>>>

# Run the following command to import the H2O module:
>>> import h2o

# Run the following command to initialize H2O on your local machine (single-node cluster).
>>> h2o.init()

# If desired, run the GLM, GBM, or Deep Learning demo
>>> h2o.demo("glm")
>>> h2o.demo("gbm")
>>> h2o.demo("deeplearning")

# Import the Iris (with headers) dataset.
>>> path = "smalldata/iris/iris_wheader.csv"
>>> iris = h2o.import_file(path=path)

# View a summary of the imported dataset.
>>> iris.summary
  sepal_len    sepal_wid    petal_len    petal_wid        class
-----------  -----------  -----------  -----------  -----------
        5.1          3.5          1.4          0.2  Iris-setosa
        4.9          3            1.4          0.2  Iris-setosa
        4.7          3.2          1.3          0.2  Iris-setosa
        4.6          3.1          1.5          0.2  Iris-setosa
        5            3.6          1.4          0.2  Iris-setosa
        5.4          3.9          1.7          0.4  Iris-setosa
        4.6          3.4          1.4          0.3  Iris-setosa
        5            3.4          1.5          0.2  Iris-setosa
        4.4          2.9          1.4          0.2  Iris-setosa
        4.9          3.1          1.5          0.1  Iris-setosa

[150 rows x 5 columns]
<bound method H2OFrame.summary of >
>>>

Getting Started with Sparkling Water¶

• Download Sparkling Water: Go here to download Sparkling Water.
• Sparkling Water Development Documentation: Read this document first to get started with Sparkling Water.
• Launch on Hadoop and Import from HDFS: Go here to learn how to start Sparkling Water on Hadoop.
• Sparkling Water Tutorials: Go here for demos and examples.
  • Sparkling Water K-means Tutorial: Go here to view a demo that uses Scala to create a K-means model.
  • Sparkling Water GBM Tutorial: Go here to view a demo that uses Scala to create a GBM model.
  • Sparkling Water on YARN: Follow these instructions to run Sparkling Water on a YARN cluster.
• Building Machine Learning Applications with Sparkling Water: This short tutorial describes project building and demonstrates the capabilities of Sparkling Water using Spark Shell to build a Deep Learning model.
• Sparkling Water FAQ: This FAQ provides answers to many common questions about Sparkling Water.
• Connecting RStudio to Sparkling Water: This illustrated tutorial describes how to use RStudio to connect to Sparkling Water.

Sparkling Water Blog Posts¶

• How Sparkling Water Brings H2O to Spark
• H2O - The Killer App on Spark
• In-memory Big Data: Spark + H2O

Sparkling Water Meetup Slide Decks¶

• Sparkling Water Meetups
• Interactive Session on Sparkling Water
• Sparkling Water Hands-On
• Additional Sparkling Water Meetup meeting notes

PySparkling¶

Note: PySparkling requires Sparkling Water 1.6 or later.

H2O’s PySparkling package is not available through pip. (There is another similarly-named package.) H2O’s PySparkling package requires EasyInstall.

To install H2O’s PySparkling package, use the egg file included in the distribution.

1. Download Spark 1.6.
2. Set the SPARK_HOME and MASTER variables as described on the Downloads page.
3. Download Sparkling Water 1.6
4. In the unpacked Sparkling Water directory, run the following command: easy_install --upgrade sparkling-water-1.6/py/dist/pySparkling-1.6-py2.7.egg

Anaconda Cloud Users¶

You can run H2O in an Anaconda Cloud environment. Conda 2.7 and 3.5 repos are supported as are a number of H2O versions. Refer to https://anaconda.org/h2oai/h2o/files to view a list of available H2O versions. Anaconda users can refer to the Install on Anaconda Cloud section for information about installing H2O in an Anaconda Cloud.

Supported Versions¶

• CDH 5.4
• CDH 5.5
• CDH 5.6
• CDH 5.7
• CDH 5.8
• CDH 5.10
• HDP 2.2
• HDP 2.3
• HDP 2.4
• HDP 2.5
• HDP 2.6
• MapR 4.0
• MapR 5.0
• MapR 5.1
• MapR 5.2
• IOP 4.2

Important Points to Remember:

• The command used to launch H2O differs from previous versions. (Refer to the Walkthrough section.)
• Launching H2O on Hadoop requires at least 6 GB of memory
• Each H2O node runs as a mapper
• Run only one mapper per host
• There are no combiners or reducers
• Each H2O cluster must have a unique job name
• -mapperXmx, -nodes, and -output are required
• Root permissions are not required - just unzip the H2O .zip file on any single node

Prerequisite: Open Communication Paths¶

H2O communicates using two communication paths. Verify these are open and available for use by H2O.

Path 1: mapper to driver

Optionally specify this port using the -driverport option in the hadoop jar command (see “Hadoop Launch Parameters” below). This port is opened on the driver host (the host where you entered the hadoop jar command). By default, this port is chosen randomly by the operating system. If you don’t want to specify an exact port but you still want to restrict the port to a certain range of ports, you can use the option -driverportrange.

Path 2: mapper to mapper

Optionally specify this port using the -baseport option in the hadoop jar command (refer to Hadoop Launch Parameters below. This port and the next subsequent port are opened on the mapper hosts (the Hadoop worker nodes) where the H2O mapper nodes are placed by the Resource Manager. By default, ports 54321 (TCP) and 54322 (TCP & UDP) are used.

The mapper port is adaptive: if 54321 and 54322 are not available, H2O will try 54323 and 54324 and so on. The mapper port is designed to be adaptive because sometimes if the YARN cluster is low on resources, YARN will place two H2O mappers for the same H2O cluster request on the same physical host. For this reason, we recommend opening a range of more than two ports (20 ports should be sufficient).

Walkthrough¶

The following steps show you how to download or build H2O with Hadoop and the parameters involved in launching H2O from the command line.

1. Download the latest H2O release for your version of Hadoop. Refer to the H2O on Hadoop tab of the download page for either the latest stable release or the nightly bleeding edge release.

2. Prepare the job input on the Hadoop Node by unzipping the build file and changing to the directory with the Hadoop and H2O’s driver jar files.

   unzip h2o-{{project_version}}-*.zip
   cd h2o-{{project_version}}-*
   

3. To launch H2O nodes and form a cluster on the Hadoop cluster, run:

   hadoop jar h2odriver.jar -nodes 1 -mapperXmx 6g -output hdfsOutputDirName
   

   The above command launches a 6g node of H2O. We recommend you launch the cluster with at least four times the memory of your data file size.

   • mapperXmx is the mapper size or the amount of memory allocated to each node. Specify at least 6 GB.
   • nodes is the number of nodes requested to form the cluster.
   • output is the name of the directory created each time a H2O cloud is created so it is necessary for the name to be unique each time it is launched.

4. To monitor your job, direct your web browser to your standard job tracker Web UI. To access H2O’s Web UI, direct your web browser to one of the launched instances. If you are unsure where your JVM is launched, review the output from your command after the nodes has clouded up and formed a cluster. Any of the nodes’ IP addresses will work as there is no master node.

   Determining driver host interface for mapper->driver callback...
   [Possible callback IP address: 172.16.2.181]
   [Possible callback IP address: 127.0.0.1]
   ...
   Waiting for H2O cluster to come up...
   H2O node 172.16.2.184:54321 requested flatfile
   Sending flatfiles to nodes...
    [Sending flatfile to node 172.16.2.184:54321]
   H2O node 172.16.2.184:54321 reports H2O cluster size 1
   H2O cluster (1 nodes) is up
   Blocking until the H2O cluster shuts down...
   

Hadoop Launch Parameters¶

• -h | -help: Display help

• -jobname <JobName>: Specify a job name for the Jobtracker to use; the default is H2O_nnnnn (where n is chosen randomly)

• -principal <kerberos principal> -keytab <keytab path> | -run_as_user <hadoop username>: Optionally specify a Kerberos principal and keytab or specify the run_as_user parameter to start clusters on behalf of the user/principal. Note that using run_as_user implies that the Hadoop cluster does not have Kerberos.

• -driverif <IP address of mapper -> driver callback interface>: Specify the IP address for callback messages from the mapper to the driver.

• -driverport <port of mapper -> callback interface>: Specify the port number for callback messages from the mapper to the driver.

• -driverportrange <range portX-portY of mapper-> callback interface>: Specify the allowed port range of the driver callback interface, eg. 50000-55000.

• -network <IPv4Network1>[,<IPv4Network2>]: Specify the IPv4 network(s) to bind to the H2O nodes; multiple networks can be specified to force H2O to use the specified host in the Hadoop cluster. 10.1.2.0/24 allows 256 possibilities.

• -timeout <seconds>: Specify the timeout duration (in seconds) to wait for the cluster to form before failing. Note: The default value is 120 seconds; if your cluster is very busy, this may not provide enough time for the nodes to launch. If H2O does not launch, try increasing this value (for example, -timeout 600).

• -disown: Exit the driver after the cluster forms.

  Note: For Qubole users who include the -disown flag, if your cluster is dying right after launch, add -Dmapred.jobclient.killjob.onexit=false as a launch parameter.

• -notify <notification file name>: Specify a file to write when the cluster is up. The file contains the IP and port of the embedded web server for one of the nodes in the cluster. All mappers must start before the H2O cloud is considered “up”.

• -mapperXmx <per mapper Java Xmx heap size>: Specify the amount of memory to allocate to H2O (at least 6g).

• -extramempercent <0-20>: Specify the extra memory for internal JVM use outside of the Java heap. This is a percentage of mapperXmx.

• -n | -nodes <number of H2O nodes>: Specify the number of nodes.

• -nthreads <maximum number of CPUs>: Specify the number of CPUs to use. This defaults to using all CPUs on the host, or you can enter a positive integer.

• -baseport <initialization port for H2O nodes>: Specify the initialization port for the H2O nodes. The default is 54321.

• -ea: Enable assertions to verify boolean expressions for error detection.

• -verbose:gc: Include heap and garbage collection information in the logs.

• -XX:+PrintGCDetails: Include a short message after each garbage collection.

• -license <license file name>: Specify the directory of local filesytem location and the license file name.

• -o | -output <HDFS output directory>: Specify the HDFS directory for the output.

• -flow_dir <Saved Flows directory>: Specify the directory for saved flows. By default, H2O will try to find the HDFS home directory to use as the directory for flows. If the HDFS home directory is not found, flows cannot be saved unless a directory is specified using -flow_dir.

Accessing S3 Data from Hadoop¶

H2O launched on Hadoop can access S3 Data in addition to to HDFS. To enable access, follow the instructions below.

Edit Hadoop’s core-site.xml, then set the HADOOP_CONF_DIR environment property to the directory containing the core-site.xml file. For an example core-site.xml file, refer to Core-site.xml Example. Typically, the configuration directory for most Hadoop distributions is /etc/hadoop/conf.

You can also pass the S3 credentials when launching H2O with the Hadoop jar command. Use the -D flag to pass the credentials:

hadoop jar h2odriver.jar -Dfs.s3.awsAccessKeyId="${AWS_ACCESS_KEY}" -Dfs.s3n.awsSecretAccessKey="${AWS_SECRET_KEY}" -n 3 -mapperXmx 10g  -output outputDirectory

where AWS_ACCESS_KEY represents your user name and AWS_SECRET_KEY represents your password.

Then import the data with the S3 URL path:

• To import the data from the Flow API:

  importFiles [ "s3n:/path/to/bucket/file/file.tab.gz" ]
  

• To import the data from the R API:

  h2o.importFile(path = "s3n://bucket/path/to/file.csv")
  

• To import the data from the Python API:

  h2o.import_frame(path = "s3n://bucket/path/to/file.csv")
  

YARN Best Practices¶

YARN (Yet Another Resource Manager) is a resource management framework. H2O can be launched as an application on YARN. If you want to run H2O on Hadoop, essentially, you are running H2O on YARN. If you are not currently using YARN to manage your cluster resources, we strongly recommend it.

- <name>yarn.nodemanager.resource.memory-mb</name>
- <name>yarn.scheduler.maximum-allocation-mb</name>

hadoop jar h2odriver.jar -Dmapreduce.job.queuename=<my-h2o-queue> -nodes <num-nodes> -mapperXmx 6g -output hdfsOutputDirName

Prerequisites¶

• Linux kernel version 3.8+ or Mac OS X 10.6+
• VirtualBox
• Latest version of Docker is installed and configured
• Docker daemon is running - enter all commands below in the Docker daemon window
• Using User directory (not root)

Notes¶

• Older Linux kernel versions are known to cause kernel panics that break Docker; there are ways around it, but these should be attempted at your own risk. To check the version of your kernel, run uname -r at the command prompt. The walkkthrough that follows has been tested on a Mac OS X 10.10.1.
• The Dockerfile always pulls the latest H2O release.
• The Docker image only needs to be built once.

Walkthrough¶

Step 1 - Install and Launch Docker

Depending on your OS, select the appropriate installation method:

• Mac Installation
• Ubuntu Installation
• Other OS Installations

Step 2 - Create or Download Dockerfile

Note: If the following commands do not work, prepend them with sudo.

1. Create a folder on the Host OS to host your Dockerfile by running:

2. Next, either download or create a Dockerfile, which is a build recipe that builds the container.

Download and use our Dockerfile template by running:

cd /data/h2o-{{branch_name}}
wget https://raw.githubusercontent.com/h2oai/h2o-3/master/Dockerfile

The Dockerfile:

• obtains and updates the base image (Ubuntu 14.04)
• installs Java 7
• obtains and downloads the H2O build from H2O’s S3 repository
• exposes ports 54321 and 54322 in preparation for launching H2O on those ports

Step 3 - Build Docker image from Dockerfile

From the /data/h2o-{{branch_name}} directory, run:

docker build -t "h2o.ai/{{branch_name}}:v5" .

**Note**: ``v5`` represents the current version number.

Because it assembles all the necessary parts for the image, this process can take a few minutes.

Step 4 - Run Docker Build

On a Mac, use the argument -p 54321:54321 to expressly map the port 54321. This is not necessary on Linux.

docker run -ti -p 54321:54321 h2o.ai/{{branch_name}}:v5 /bin/bash

**Note**: ``v5`` represents the version number.

Step 5 - Launch H2O

Navigate to the /opt directory and launch H2O. Change the value of -Xmx to the amount of memory you want to allocate to the H2O instance. By default, H2O launches on port 54321.

cd /opt
java -Xmx1g -jar h2o.jar

Step 6 - Access H2O from the web browser or R

• On Linux: After H2O launches, copy and paste the IP address and port of the H2O instance into the address bar of your browser. In the following example, the IP is 172.17.0.5:54321.

03:58:25.963 main      INFO WATER: Cloud of size 1 formed [/172.17.0.5:54321 (00:00:00.000)]

• On OSX: Locate the IP address of the Docker’s network (192.168.59.103 in the following examples) that bridges to your Host OS by opening a new Terminal window (not a bash for your container) and running boot2docker ip.

$ boot2docker ip
192.168.59.103

You can also view the IP address (192.168.99.100 in the example below) by scrolling to the top of the Docker daemon window:

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/


docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com

After obtaining the IP address, point your browser to the specified ip address and port. In R, you can access the instance by installing the latest version of the H2O R package and running:

library(h2o)
dockerH2O <- h2o.init(ip = "192.168.59.103", port = 54321)