Import & Export H2O Frames from/to S3
-------------------------------------

Sparkling Water car read and write H2O frames from and to S3. Several configuration steps are
required.

Specify the AWS Dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To enable support for S3A/S3N, we need to start Sparkling Water with the following extra dependencies:

- ``org.apache.hadoop:hadoop-aws:2.7.3``
- ``spark.jars.packages com.amazonaws:aws-java-sdk:1.7.4``

For production environments, we advise to download these jars and add them on your Spark path manually by copying them to
``$SPARK_HOME/jars`` directory.

Additionally, we can also use the ``--packages`` option when starting Sparkling Water as:

 .. code:: bash

    ./bin/sparkling-shell --packages spark.jars.packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3

The same holds for PySparkling. We can also add the following line to the spark-defaults.conf file:

 .. code:: bash

    spark.jars.packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3

This line ensures that we don't need to specify the ``--packages`` option all the time.

Configuring S3A
~~~~~~~~~~~~~~~
In order to read and write to S3A, please add the following lines to the ``spark-defaults.conf`` file
in the ``$SPARK_HOME/conf`` directory:

 .. code:: bash

     spark.hadoop.fs.s3a.impl                org.apache.hadoop.fs.s3a.S3AFileSystem
     spark.hadoop.fs.s3a.access.key          {{AWS_ACCESS_KEY}}
     spark.hadoop.fs.s3a.secret.key          {{AWS_SECRET_KEY}}

where ``{{AWS_ACCESS_KEY}}`` should be substituted by AWS access key and ``{{AWS_SECRET_KEY}}`` by
AWS secret key.

Configuring S3N
~~~~~~~~~~~~~~~
In order to read and write to S3N, please add the following lines to the ``spark-defaults.conf`` file
in the ``$SPARK_HOME/conf`` directory:

 .. code:: bash

    spark.hadoop.fs.s3n.impl                org.apache.hadoop.fs.s3native.NativeS3FileSystem
    spark.hadoop.fs.s3n.awsAccessKeyId      {{AWS_ACCESS_KEY}}
    spark.hadoop.fs.s3n.awsSecretAccessKey  {{AWS_SECRET_KEY}}

where ``{{AWS_ACCESS_KEY}}`` should be substituted by AWS access key and ``{{AWS_SECRET_KEY}}`` by
AWS secret key.


Sparkling Water Example Code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When you configured Sparkling Water as explained above, please start Sparkling Shell as

 .. code:: bash

    ./bin/sparkling-shell


Next, let's start ``H2OContext``. This context brings H2O support into Spark environment:

 .. code:: scala

    import ai.h2o.sparkling._
    val hc = H2OContext.getOrCreate()

Finally, read the data:

 .. code:: scala

    import java.net.URI
    val fr = H2OFrame(new URI("s3n://data.h2o.ai/h2o-open-tour/2016-nyc/weather.csv"))

PySparkling Example Code
~~~~~~~~~~~~~~~~~~~~~~~~

When you configured PySparkling as explained above, please start PySparkling as

 .. code:: python

    ./bin/pysparkling

Next, let's start ``H2OContext``. This context brings H2O support into Spark environment:

 .. code:: python

    from pysparkling import *
    hc = H2OContext.getOrCreate()

Finally, read the data:

 .. code:: python

    fr = h2o.import_file("s3n://data.h2o.ai/h2o-open-tour/2016-nyc/weather.csv")

In PySparkling, you can also export the file to S3 as:

 .. code:: python

    h2o.export_file("s3n://path/to/target/location")