Running Sparkling Water Examples

The Sparkling Water distribution includes also a set of examples. You can find their implementation in examples directory. You can build and run them in the following way:

  1. Build a package that can be submitted to Spark cluster:

    ./gradlew build -x check
    
  2. Set the configuration of the demo Spark cluster (for example, local[*])

    export SPARK_HOME="/path/to/spark/installation"
    export MASTER="local[*]"
    

    In this example, the description local[*] causes creation of a single node local cluster.

  3. And run the example:

  • On Local Cluster:

    The local cluster is defined by MASTER address local, local[*] or additional variants available at Spark Master URLs.

    bin/run-example.sh <name of example>
    
  • On a Spark Standalone Cluster:

    • Run the Spark cluster, for example via:

      bin/launch-spark-cloud.sh
      
    • Verify that Spark is running: The Spark UI on http://localhost:8080/ should show 3 worker nodes

    • Export MASTER address of Spark master using:

      export MASTER="spark://localhost:7077"
      
    • Run example:

      bin/run-example.sh <name of example>
      
    • Observe the status of the application via Spark UI on http://localhost:8080/

Additional Details and Examples

For more information about examples or to view additional examples, review the README file in the examples folder.