Security¶

Terms¶

Assumptions (Threat Model)¶

1. H2O lives in a secure data center.
2. Denial of service is not a concern.
   • H2O is not designed to withstand a DOS attack.
3. HTTP traffic between the user client and H2O cluster needs to be encrypted.
   • This is true for both interactive sessions (e.g the H2O Flow Web UI) and programmatic sessions (e.g. an R program).
4. Man-in-the-middle attacks are of low concern.
   • Certificate checking on the client side for R/python is not yet implemented.
5. Internal binary H2O node-to-H2O node traffic does not need to be secured.
   • The customer is responsible for the H2O cluster’s perimeter security if this is a concern.
   • An example would be putting the nodes for an H2O cluster in a VLAN and opening up one port so user clients can reach the H2O cluster on the embedded web port.
6. You trust the person that starts H2O to start it correctly.
   • Enabling H2O security requires specifying the correct security options.
7. User client sessions do not need to expire. A session lives at most as long as the cluster lifetime. H2O clusters are started and stopped “frequently enough”.
   • All data is stored in-memory, so restarting the H2O cluster wipes all data from memory, and there is nothing to clean from disk.
8. Once a user is authenticated for access to H2O, they have full access.
   • H2O supports authentication but not authorization or access control (ACLs).
9. H2O clusters are meant to be accessed by only one user.
   • Each user starts their own H2O cluster.
   • H2O only allows access to the embedded web port to the person that started the cluster.

Data Chain-of-Custody in a Hadoop Data Center Environment¶

Note: This holds true for all versions of Hadoop (including YARN) supported by H2O.

Through this sequence, it is shown that a user is only able to access the same data from H2O that they could already access from normal Hadoop jobs.

1. Data lives in HDFS
2. The files in HDFS have permissions
3. An HDFS user has permissions (capabilities) to access certain files
4. Kerberos (kinit) can be used to authenticate a user in a Hadoop environment
5. A user’s Hadoop MapReduce job inherits the permissions (capabilities) of the user, as well as kinit metadata
6. H2O is a Hadoop MapReduce job
7. H2O can only access the files in HDFS that the user has permission to access
8. Only the user that started the cluster is authenticated for access to the H2O cluster
9. The authenticated user can access the same data in H2O that he could access via HDFS

What is being Secured Today¶

1. Standard file permissions security is provided by the Operating System and by HDFS.
2. The embedded web port in each node of H2O can be secured in two ways:

Method	Description
HTTPS	Encrypted socket communication between the user client and the embedded H2O web port.
Authentication	An HTTP Basic Auth username and password from the user client.

Note: Embedded web port HTTPS and authentication may be used separately or together.

What is Specifically not being Secured Today¶

• Internal H2O node-to-H2O node communication.

Standalone H2O¶

Since H2O is a regular Java program, the files H2O can access are restricted by the user’s Operating System permissions (capabilities).

H2O on Hadoop¶

Since H2O is a regular Hadoop MapReduce program, the files H2O can access are restricted by the standard HDFS permissions of the user that starts H2O.

Since H2O is a regular Hadoop MapReduce program, Kerberos (kinit) works seamlessly. (No code was added to H2O to support Kerberos.)

Sparkling Water on YARN¶

Similar to H2O on Hadoop, this configuration is H2O on Spark on YARN. The YARN job inherits the HDFS permissions of the user.

HTTPS¶

h2o.init(ip = "a.b.c.d", port = 54321, https = TRUE, insecure = TRUE)

-jks <filename>
     Java keystore file

-jks_pass <password>
     (Default is 'h2oh2o')

java -jar h2o.jar -jks h2o.jks

-jks <filename>
     Java keystore file

-jks_pass <password>
     (Default is 'h2oh2o')

hadoop jar h2odriver.jar -n 3 -mapperXmx 10g -jks h2o.jks -output hdfsOutputDirectory

$SPARK_HOME/bin/spark-submit --class water.SparklingWaterDriver --conf spark.ext.h2o.jks=/path/to/h2o.jks sparkling-water-assembly-0.2.17-SNAPSHOT-all.jar

# Be paranoid and delete any previously existing keystore.
rm -f mykeystore.jks

# Generate a new keystore.
keytool -genkey -keyalg RSA -keystore mykeystore.jks -storepass mypass -keysize 2048
What is your first and last name?
  [Unknown]:
What is the name of your organizational unit?
  [Unknown]:
What is the name of your organization?
  [Unknown]:
What is the name of your City or Locality?
  [Unknown]:
What is the name of your State or Province?
  [Unknown]:
What is the two-letter country code for this unit?
  [Unknown]:
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
  [no]:  yes

Enter key password for <mykey>
    (RETURN if same as keystore password):

# Run H2O using the newly generated self-signed keystore.
java -jar h2o.jar -jks mykeystore.jks -jks_pass mypass

Kerberos Authentication¶

h2o.init(ip = "a.b.c.d", port = 54321, username = "myusername", password = "mypassword")

h2o.init(ip = "a.b.c.d", port = 54321, username = "myusername", password = "mypassword")

krb5loginmodule {
     com.sun.security.auth.module.Krb5LoginModule required
     java.security.krb5.realm="0XDATA.LOC"
     java.security.krb5.kdc="ldap.0xdata.loc";
};

-kerberos_login
      Use Jetty KerberosLoginService

-login_conf <filename>
      LoginService configuration file

-user_name <username>
      Override name of user for which access is allowed

java -jar h2o.jar -kerberos_login -login_conf kerb.conf -user_name kerb_principal

java -Djava.security.krb5.realm="0XDATA.LOC" -Djava.security.krb5.kdc="ldap.0xdata.loc" -jar h2o.jar -kerberos_login -login_conf kerb.conf -user_name kerb_principal

-kerberos_login
      Use Jetty KerberosLoginService

-login_conf <filename>
      LoginService configuration file

-user_name <username>
      Override name of user for which access is allowed

hadoop jar h2odriver.jar -n 3 -mapperXmx 10g -kerberos_login -login_conf kerb.conf -output hdfsOutputDirectory -user_name kerb_principal

$SPARK_HOME/bin/spark-submit --class water.SparklingWaterDriver --conf spark.ext.h2o.kerberos.login=true --conf spark.ext.h2o.user.name=kerb_principal --conf spark.ext.h2o.login.conf=kerb.conf sparkling-water-assembly-0.2.17-SNAPSHOT-all.jar

LDAP Authentication¶

H2O client and server side configuration for LDAP is discussed below. Authentication is implemented using Basic Auth.

h2o.init(ip = "a.b.c.d", port = 54321, username = "myusername", password = "mypassword")

ldaploginmodule {
    org.eclipse.jetty.plus.jaas.spi.LdapLoginModule required
    debug="true"
    useLdaps="false"
    contextFactory="com.sun.jndi.ldap.LdapCtxFactory"
    hostname="ldap.0xdata.loc"
    port="389"
    bindDn="cn=admin,dc=0xdata,dc=loc"
    bindPassword="0xdata"
    authenticationMethod="simple"
    forceBindingLogin="true"
    userBaseDn="ou=users,dc=0xdata,dc=loc"
    userRdnAttribute="uid"
    userIdAttribute="uid"
    userPasswordAttribute="userPassword"
    userObjectClass="inetOrgPerson"
    roleBaseDn="ou=groups,dc=0xdata,dc=loc"
    roleNameAttribute="cn"
    roleMemberAttribute="uniqueMember"
    roleObjectClass="groupOfUniqueNames";
};

-ldap_login
      Use Jetty LdapLoginService

-login_conf <filename>
      LoginService configuration file

-user_name <username>
      Override name of user for which access is allowed

java -jar h2o.jar -ldap_login -login_conf ldap.conf

java -jar h2o.jar -ldap_login -login_conf ldap.conf -user_name myLDAPusername

-ldap_login
      Use Jetty LdapLoginService

-login_conf <filename>
      LoginService configuration file

-user_name <username>
      Override name of user for which access is allowed

hadoop jar h2odriver.jar -n 3 -mapperXmx 10g -ldap_login -login_conf ldap.conf -output hdfsOutputDirectory

hadoop jar h2odriver.jar -n 3 -mapperXmx 10g -ldap_login -login_conf ldap.conf -user_name myLDAPusername -output hdfsOutputDirectory

$SPARK_HOME/bin/spark-submit --class water.SparklingWaterDriver --conf spark.ext.h2o.ldap.login=true --conf spark.ext.h2o.login.conf=/path/to/ldap.conf sparkling-water-assembly-0.2.17-SNAPSHOT-all.jar

$SPARK_HOME/bin/spark-submit --class water.SparklingWaterDriver --conf spark.ext.h2o.ldap.login=true --conf spark.ext.h2o.user.name=myLDAPusername --conf spark.ext.h2o.login.conf=/path/to/ldap.conf sparkling-water-assembly-0.2.17-SNAPSHOT-all.jar

Hash File Authentication¶

H2O client and server side configuration for a hardcoded hash file is discussed below. Authentication is implemented using Basic Auth.

h2o.init(ip = "a.b.c.d", port = 54321, username = "myusername", password = "mypassword")

# See https://wiki.eclipse.org/Jetty/Howto/Secure_Passwords
# java -cp h2o.jar org.eclipse.jetty.util.security.Password
username1: password1
username2: MD5:6cb75f652a9b52798eb6cf2201057c73

java -cp h2o.jar org.eclipse.jetty.util.security.Password username password

-hash_login
      Use Jetty HashLoginService

-login_conf <filename>
      LoginService configuration file

java -jar h2o.jar -hash_login -login_conf realm.properties

-hash_login
      Use Jetty HashLoginService

-login_conf <filename>
      LoginService configuration file

hadoop jar h2odriver.jar -n 3 -mapperXmx 10g -hash_login -login_conf realm.propertes -output hdfsOutputDirectory

$SPARK_HOME/bin/spark-submit --class water.SparklingWaterDriver --conf spark.ext.h2o.hash.login=true --conf spark.ext.h2o.login.conf=/path/to/realm.properties sparkling-water-assembly-0.2.17-SNAPSHOT-all.jar

Usage¶

hadoop jar h2odriver.jar -nodes 4 -mapperXmx 6g -output hdfsOutputDirName -internal_secure_connections

hadoop jar h2odriver.jar -nodes 4 -mapperXmx 6g -output hdfsOutputDirName -internal_security security.properties

Configuration¶

To enable this feature, set the -internal_security parameter when starting an H2O node, and point that to a configuration file (key=value format) that contains the following values:

• h2o_ssl_jks_internal (required): The path (absolute or relative) to the key-store file used for internal SSL communication
• h2o_ssl_jks_password (required): The password for the internal key-store
• h2o_ssl_jts_internal (optional): The path (absolute or relative) to the trust-store file used for internal SSL communication. If not present, then h2o_ssl_jks_internal will be used.
• h2o_ssl_jts_password (optional): The password to the internal trust-store. If not present, then h2o_ssl_jks_password will be used.
• h2o_ssl_protocol (optional): The protocol name used during encrypted communication (supported by JVM). This defaults to TSLv1.2.
• h2o_ssl_enabled_algorithms (optional): A comma separated list of enabled cipher algorithms. Include only those that are supported by JVM.

This must be set for every node in the cluster. Every node needs to have access to both Java keystore and Java truststore containing appropriate keys and certificates.

This feature should not be used together with the -useUDP flag, as we currently do not support UDP encryption through DTLS or any other protocol that might result in unencrypted data transfers.

Keystore/Truststore Generation¶

Keystore/truststore creation and distribution are deployment specific and have to be handled by the end user.

Basic keystore/truststore generation can be done using the keytool program, which ships with Java, documentation can be found here. Each node should have a key pair generated, and all public keys should be imported into a single truststore, which should be distributed to all the nodes.

The simplest (though not recommended) way would be to call:

keytool -genkeypair -keystore h2o-internal.jks -alias h2o-internal

Then distribute the h2o-internal.jks file to all the nodes, and set it as both the keystore and truststore in ssl.config.

A more secure way would be to:

1. Run the same command on each node:

keytool -genkeypair -keystore h2o-internal.jks -alias h2o-internal

2. Extract the certificate on each node:

keytool -export -keystore h2o-internal.jks -alias signFiles -file node<number>.cer

3. Distribute all of the above certificates to each node, and on each node create a truststore containing all of them (or put all certificates on one node, import to truststore and distribute that truststore to each node):

keytool -importcert -file node<number>.cer -keystore truststore.jks -alias node<number>

Performance¶

Turning on SSL may result in performance overhead for settings and algorithms that exchange data between nodes due to encryption/decryption time. Some algorithms might also slower because of this.

Example benchmark on a 5 node cluster (6GB memory per node) working with a 5.8mln row dataset (580MB):

Caveats and Missing Pieces¶

• This feature CANNOT be used together with the -useUDP flag. We currently do not support DTLS or any other encryption for UDP.
• Should you start a mixed cloud of SSL and nonSSL nodes, the SSL ones will fail to bootstrap, while the nonSSL ones will become unresponsive.
• H2O does not provide in-memory data encryption. This might spill data to disk in unencrypted form should swaps to disk occur. As a workaround, an encrypted drive is advised.
• H2O does not support encryption of data saved to disk, should appropriate flags be enabled. Similar to the previous caveat, the user can use an encrypted drive to work around this issue.
• H2O supports only SSL and does not support SASL.