Trait/Object

water.support

SparkContextSupport

Related Docs: object SparkContextSupport | package support

Permalink

trait SparkContextSupport extends AnyRef

Useful methods to configure Spark context

Linear Supertypes
AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkContextSupport
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def absPath(path: String): String

    Permalink

    Return absolute path of a file location

    Return absolute path of a file location

    path

    path to a file

    returns

    absolute path to a file

  5. def addFiles(spark: SparkSession, files: String*): Unit

    Permalink

    Add files into Spark context

    Add files into Spark context

    spark

    Spark session

    files

    path to files to add

  6. def addFiles(sc: SparkContext, files: String*): Unit

    Permalink

    Add files into Spark context

    Add files into Spark context

    sc

    Spark context

    files

    path to files to add

  7. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  8. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def configure(appName: String = "Sparkling Water Demo", defaultMaster: String = "local[*]"): SparkConf

    Permalink

    Helper method to configure basic spark settings

    Helper method to configure basic spark settings

    appName

    application name

    defaultMaster

    master string

    returns

    Spark configuration

  10. def enforceLocalSparkFile(file: String): String

    Permalink

    This method enforces the local path.

    This method enforces the local path. For example, if running on Hadoop, Spark by default uses HDFS. If we need to access a local file, for example, from driver, then this method might be useful

    file

    path to the file

    returns

    local path to the file

  11. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  13. def exportSparkModel(model: Any, destination: URI): Unit

    Permalink

    Export Spark Model to a specified destination URI

    Export Spark Model to a specified destination URI

    model

    Spark model

    destination

    destination URI

  14. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  15. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  16. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  17. def isFileDistributed(sc: SparkContext, filePath: String): Boolean

    Permalink

    Returns true if the file has already been added to Spark files

    Returns true if the file has already been added to Spark files

    sc

    Spark context

    filePath

    any path containing the file name

    returns

    true if the file has already been added to Spark files, otherwise false

  18. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  19. def loadSparkModel[M](source: URI): M

    Permalink

    Import Spark Model from a specified source URI

    Import Spark Model from a specified source URI

    M

    Model Type

    source

    source URI

    returns

    the loaded model

  20. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  21. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  22. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  23. def sparkContext(conf: SparkConf): SparkContext

    Permalink

    Get or create spark context

    Get or create spark context

    conf

    spark configuration

    returns

    Spark context

  24. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  25. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  26. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped