Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def clone(): AnyRef

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @native() @throws( ... )

6.  def corr(dataset: Dataset[_], column: String): DataFrame

   Compute the Pearson correlation matrix for the input Dataset of Vectors.

   Compute the Pearson correlation matrix for the input Dataset of Vectors.

   Annotations

   @Since( "2.2.0" )

7.  def corr(dataset: Dataset[_], column: String, method: String): DataFrame

   :: Experimental :: Compute the correlation matrix for the input Dataset of Vectors using the specified method.

   :: Experimental :: Compute the correlation matrix for the input Dataset of Vectors using the specified method. Methods currently supported: pearson (default), spearman.

   dataset

   A dataset or a dataframe

   column

   The name of the column of vectors for which the correlation coefficient needs to be computed. This must be a column of the dataset, and it must contain Vector objects.

   method

   String specifying the method to use for computing correlation. Supported: pearson (default), spearman

   returns

   A dataframe that contains the correlation matrix of the column of vectors. This dataframe contains a single row and a single column of name '$METHODNAME($COLUMN)'.

   Annotations

   @Since( "2.2.0" )

   Exceptions thrown

   if the column is not a valid column in the dataset, or if the content of this column is not of type Vector. Here is how to access the correlation coefficient:

   val data: Dataset[Vector] = ...
   val Row(coeff: Matrix) = Correlation.corr(data, "value").head
   // coeff now contains the Pearson correlation matrix.

   Note

   For Spearman, a rank correlation, we need to create an RDD[Double] for each column and sort it in order to retrieve the ranks and then join the columns back into an RDD[Vector], which is fairly costly. Cache the input Dataset before calling corr with method = "spearman" to avoid recomputing the common lineage.

8.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

9.  def equals(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

10.  def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

11.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

12.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

13.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

14.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

15.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

16.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

17.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

18.  def toString(): String

    Definition Classes

    AnyRef → Any

19.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

20.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

21.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @native() @throws( ... )