Use Sparkling Water with Amazon EMR from the Edge Node¶
Sparkling Water can be deployed on a cluster on EMR or directly on EC2 machines. The jobs can be submitted from the edge node which has several security and other benefits.
In order to start Sparkling Water from the Edge node (for example, different EC2 machine), please make sure that:
- You are able to submit Spark Jobs from the edge node first.
- You are starting Sparkling Water with
spark.ext.h2o.client.ignore.SPARK_PUBLIC_DNS=trueconfiguration. This configuration ensures that the H2O client will ignore
SPARK_PUBLIC_DNSconfiguration on the client side which is required for the correct cloud formation. The
SPARK_PUBLIC_DNSis still being applied as expected to all Spark services.
- Make sure that
DNS hostnamesare set to
truein the AWS DNS configuration. If you can’t set these option to
true, AWS provided DNS server will not be able to resolve internal DNS names. In that case, please start Sparkling Water with additional option
spark.ext.h2o.ip.based.flatfile=true, which ensures that all Sparkling Water worker nodes prefer to use IP addresses instead of unresolvable hostnames.