Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark

    Core Spark functionality.

    Core Spark functionality. org.apache.spark.SparkContext serves as the main entry point to Spark, while org.apache.spark.rdd.RDD is the data type representing a distributed collection, and provides most parallel operations.

    In addition, org.apache.spark.rdd.PairRDDFunctions contains operations available only on RDDs of key-value pairs, such as groupByKey and join; org.apache.spark.rdd.DoubleRDDFunctions contains operations available only on RDDs of Doubles; and org.apache.spark.rdd.SequenceFileRDDFunctions contains operations available on RDDs that can be saved as SequenceFiles. These operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] through implicit conversions.

    Java programmers should reference the org.apache.spark.api.java package for Spark programming APIs in Java.

    Classes and methods marked with Experimental are user-facing features which have not been officially adopted by the Spark project. These are subject to change or removal in minor releases.

    Classes and methods marked with Developer API are intended for advanced users want to extend Spark through lower level interfaces. These are subject to changes or removal in minor releases.

    Definition Classes
    apache
  • package ml

    DataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines.

    DataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines.

    Definition Classes
    spark
  • package image
    Definition Classes
    ml
  • ImageSchema
o

org.apache.spark.ml.image

ImageSchema

object ImageSchema

:: Experimental :: Defines the image schema and methods to read and manipulate images.

Annotations
@Experimental() @Since( "2.3.0" )
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ImageSchema
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  6. val columnSchema: StructType

    Schema for the image column: Row(String, Int, Int, Int, Int, Array[Byte])

  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def getData(row: Row): Array[Byte]

    Gets the image data

    Gets the image data

    returns

    The image data

  12. def getHeight(row: Row): Int

    Gets the height of the image

    Gets the height of the image

    returns

    The height of the image

  13. def getMode(row: Row): Int

    Gets the OpenCV representation as an int

    Gets the OpenCV representation as an int

    returns

    The OpenCV representation as an int

  14. def getNChannels(row: Row): Int

    Gets the number of channels in the image

    Gets the number of channels in the image

    returns

    The number of channels in the image

  15. def getOrigin(row: Row): String

    Gets the origin of the image

    Gets the origin of the image

    returns

    The origin of the image

  16. def getWidth(row: Row): Int

    Gets the width of the image

    Gets the width of the image

    returns

    The width of the image

  17. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  18. val imageFields: Array[String]
  19. val imageSchema: StructType

    DataFrame with a single column of images named "image" (nullable)

  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. val javaOcvTypes: Map[String, Int]

    (Java-specific) OpenCV type mapping supported

  22. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. val ocvTypes: Map[String, Int]

    (Scala-specific) OpenCV type mapping supported

  26. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  27. def toString(): String
    Definition Classes
    AnyRef → Any
  28. val undefinedImageType: String
  29. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Deprecated Value Members

  1. def readImages(path: String, sparkSession: SparkSession, recursive: Boolean, numPartitions: Int, dropImageFailures: Boolean, sampleRatio: Double, seed: Long): DataFrame

    Read the directory of images from the local or remote source

    Read the directory of images from the local or remote source

    path

    Path to the image directory

    sparkSession

    Spark Session, if omitted gets or creates the session

    recursive

    Recursive path search flag

    numPartitions

    Number of the DataFrame partitions, if omitted uses defaultParallelism instead

    dropImageFailures

    Drop the files that are not valid images from the result

    sampleRatio

    Fraction of the files loaded

    returns

    DataFrame with a single column "image" of images; see ImageSchema for the details

    Annotations
    @deprecated
    Deprecated

    (Since version 2.4.0)

    Note

    If multiple jobs are run in parallel with different sampleRatio or recursive flag, there may be a race condition where one job overwrites the hadoop configs of another.

    ,

    If sample ratio is less than 1, sampling uses a PathFilter that is efficient but potentially non-deterministic.

  2. def readImages(path: String): DataFrame

    Read the directory of images from the local or remote source

    Read the directory of images from the local or remote source

    path

    Path to the image directory

    returns

    DataFrame with a single column "image" of images; see ImageSchema for the details

    Annotations
    @deprecated
    Deprecated

    (Since version 2.4.0)

    Note

    If multiple jobs are run in parallel with different sampleRatio or recursive flag, there may be a race condition where one job overwrites the hadoop configs of another.

    ,

    If sample ratio is less than 1, sampling uses a PathFilter that is efficient but potentially non-deterministic.

Inherited from AnyRef

Inherited from Any

Members