Running RSparkling on Databricks Azure Cluster

Sparkling Water, PySparkling and RSparkling can be used on top of Databricks Azure Cluster. This tutorial is the RSparkling.

For Scala Sparkling Water, please visit Running Sparkling Water on Databricks Azure Cluster and for PySparkling, please visit Running PySparkling on Databricks Azure Cluster.

To start Sparkling Water H2OContext on Databricks Azure, the steps are:

1. Login into Microsoft Azure Portal

2. Create Databricks Azure Environment

   In order to connect to Databricks from Azure, please make sure you have created user inside Azure Active Directory and using that user for the Databricks Login.

3. Create the cluster

   • For Sparkling Water 2.3.31 select Spark 2.3.3

   It is advised to always use the latest Sparkling Water and Spark version for the given Spark major version.

   

4. Create R notebook and attach it to the created cluster. To start H2OContext, the init part of the notebook should be:

   # Install Sparklyr
   install.packages("sparklyr")
   
   # Install RSparkling 2.3.31
   install.packages("rsparkling", type = "source", repos = "http://h2o-release.s3.amazonaws.com/sparkling-water/rel-2.3/31/R")
   
   # Install H2O 3.24.0.5 (yates)
   install.packages("h2o", type = "source", repos = "http://h2o-release.s3.amazonaws.com/h2o/rel-yates/5/R")
   
   # Connect to Spark on Databricks
   library(rsparkling)
   library(sparklyr)
   sc <- spark_connect(method = "databricks")
   
   # Start H2O context
   h2o_context(sc)
   

6. And voila, we should have H2OContext running

   

   Please note that accessing H2O Flow is not currently supported on Azure Databricks.