Running Sparkling Water Examples

The Sparkling Water distribution includes also a set of examples. You can find their implementation in examples directory. You can build and run them in the following way:

1. Build a package that can be submitted to Spark cluster:

   ./gradlew build -x check
   

2. Set the configuration of the demo Spark cluster (for example, local[*])

   export SPARK_HOME="/path/to/spark/installation"
   export MASTER="local[*]"
   

   In this example, the description local[*] causes creation of a single node local cluster.

3. And run the example:

• On Local Cluster:

  The local cluster is defined by MASTER address local, local[*] or additional variants available at Spark Master URLs.

  bin/run-example.sh <name of example>
  

• On a Spark Standalone Cluster:

  • Run the Spark cluster, for example via:

    bin/launch-spark-cloud.sh
    

  • Verify that Spark is running: The Spark UI on http://localhost:8080/ should show 3 worker nodes

  • Export MASTER address of Spark master using:

    export MASTER="spark://localhost:7077"
    

  • Run example:

    bin/run-example.sh <name of example>
    

  • Observe the status of the application via Spark UI on http://localhost:8080/

Additional Details and Examples

For more information about examples or to view additional examples, review the README file in the examples folder.