EC2 Instances & S3 storage¶

Standalone Instance¶

When running H2O in standalone mode using the simple Java launch command, we can pass in the S3 credentials in two ways.

• You can pass in credentials in standalone mode by creating a core-site.xml file and pass it in with the flag -hdfs_config. For an example core-site.xml file, refer to Core-site.xml.

  1. Edit the properties in the core-site.xml file to include your Access Key ID and Access Key as shown in the following example:

  <property>
          <name>fs.s3n.awsAccessKeyId</name>
          <value>[AWS SECRET KEY]</value>
  </property>
  
  <property>
          <name>fs.s3n.awsSecretAccessKey</name>
          <value>[AWS SECRET ACCESS KEY]</value>
  </property>
  

  2. Launch with the configuration file core-site.xml by entering the following in the command line:

  java -jar h2o.jar -hdfs_config core-site.xml
  

  3. Import the data using importFile with the S3 url path: s3n://bucket/path/to/file.csv

• You can pass the AWS Access Key and Secret Access Key in an S3N Url in Flow, R, or Python (where AWS_ACCESS_KEY represents your user name and AWS_SECRET_KEY represents your password).

  • To import the data from the Flow API:

  importFiles [ "s3n://<AWS_ACCESS_KEY>:<AWS_SECRET_KEY>@bucket/path/to/file.csv" ]
  

  • To import the data from the R API:

  h2o.importFile(path = "s3n://<AWS_ACCESS_KEY>:<AWS_SECRET_KEY>@bucket/path/to/file.csv")
  

  • To import the data from the Python API:

  h2o.import_frame(path = "s3n://<AWS_ACCESS_KEY>:<AWS_SECRET_KEY>@bucket/path/to/file.csv")
  

Multi-Node Instance¶

Python and the boto Python library are required to launch a multi-node instance of H2O on EC2. Confirm these dependencies are installed before proceeding.

For more information, refer to the H2O EC2 repo.

Build a cluster of EC2 instances by running the following commands on the host that can access the nodes using a public DNS name.

1. Edit h2o-cluster-launch-instances.py to include your SSH key name and security group name, as well as any other environment-specific variables.

./h2o-cluster-launch-instances.py
./h2o-cluster-distribute-h2o.sh

—OR–

./h2o-cluster-launch-instances.py
./h2o-cluster-download-h2o.sh

Note: The second method may be faster than the first because download pulls from S3.

2. Distribute the credentials using ./h2o-cluster-distribute-aws-credentials.sh.

Note: If you are running H2O using an IAM role, it is not necessary to distribute the AWS credentials to all the nodes in the cluster. The latest version of H2O can access the temporary access key.

Caution: Distributing the AWS credentials copies the Amazon AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to the instances to enable S3 and S3N access. Use caution when adding your security keys to the cloud.

3. Start H2O by launching one H2O node per EC2 instance:

        ./h2o-cluster-start-h2o.sh

Wait 60 seconds after entering the command before entering it on the next node.

4. In your internet browser, substitute any of the public DNS node addresses for IP_ADDRESS in the following example: http://IP_ADDRESS:54321

• To start H2O: ./h2o-cluster-start-h2o.sh
• To stop H2O: ./h2o-cluster-stop-h2o.sh
• To shut down the cluster, use your Amazon AWS console to shut down the cluster manually.

Note: To successfully import data, the data must reside in the same location on all nodes.

Core-site.xml Example¶

The following is an example core-site.xml file:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <!--
    <property>
    <name>fs.default.name</name>
    <value>s3n://<your s3 bucket></value>
    </property>
    -->

    <property>
        <name>fs.s3n.awsAccessKeyId</name>
        <value>insert access key here</value>
    </property>

    <property>
        <name>fs.s3n.awsSecretAccessKey</name>
        <value>insert secret key here</value>
    </property>
    </configuration>

Launching H2O¶

Note: Before launching H2O on an EC2 cluster, verify that ports 54321 and 54322 are both accessible by TCP and UDP.

Selecting the Operating System and Virtualization Type¶

Select your operating system and the virtualization type of the prebuilt AMI on Amazon. If you are using Windows, you will need to use a hardware-assisted virtual machine (HVM). If you are using Linux, you can choose between para-virtualization (PV) and HVM. These selections determine the type of instances you can launch.

For more information about virtualization types, refer to Amazon.

Downloading Java and H2O¶

1. Download Java (JDK 1.7 or later) if it is not already available on the instance.

2. To download H2O, run the wget command with the link to the zip file available on our website by copying the link associated with the Download button for the selected H2O build.

   wget http://h2o-release.s3.amazonaws.com/h2o/{{branch_name}}/{{build_number}}/index.html
   unzip h2o-{{project_version}}.zip
   cd h2o-{{project_version}}
   java -Xmx4g -jar h2o.jar
   

3. From your browser, navigate to <Private_IP_Address>:54321 or <Public_DNS>:54321 to use H2O’s web interface.