Spark 3.0.0 ScalaDoc - org.apache.spark.mllib.clustering.DistributedLDAModel

final def !=(arg0: Any): Boolean

Definition Classes: AnyRef → Any

final def ##(): Int

Definition Classes: AnyRef → Any

final def ==(arg0: Any): Boolean

Definition Classes: AnyRef → Any

final def asInstanceOf[T0]: T0

Definition Classes: Any

def clone(): AnyRef

Attributes: protected[java.lang]
Definition Classes: AnyRef
Annotations: @native() @throws( ... )

def describeTopics(maxTermsPerTopic: Int): Array[(Array[Int], Array[Double])]

Return the topics described by weighted terms.

maxTermsPerTopic: Maximum number of terms to collect for each topic.
returns: Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.

Definition Classes: DistributedLDAModel → LDAModel
Annotations: @Since( "1.3.0" )

def describeTopics(): Array[(Array[Int], Array[Double])]

Return the topics described by weighted terms.

WARNING: If vocabSize and k are large, this can return a large object!

returns: Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.

Definition Classes: LDAModel
Annotations: @Since( "1.3.0" )

val docConcentration: Vector

Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").

This is the parameter to a Dirichlet distribution.

Definition Classes: DistributedLDAModel → LDAModel
Annotations: @Since( "1.5.0" )

final def eq(arg0: AnyRef): Boolean

Definition Classes: AnyRef

def equals(arg0: Any): Boolean

Definition Classes: AnyRef → Any

def finalize(): Unit

Attributes: protected[java.lang]
Definition Classes: AnyRef
Annotations: @throws( classOf[java.lang.Throwable] )

def formatVersion: String

Current version of model save/load format.

Attributes: protected
Definition Classes: DistributedLDAModel → Saveable

val gammaShape: Double

Shape parameter for random initialization of variational parameter gamma.

Shape parameter for random initialization of variational parameter gamma. Used for variational inference for perplexity and other test-time computations.

Attributes: protected[org.apache.spark.mllib.clustering]
Definition Classes: DistributedLDAModel → LDAModel

final def getClass(): Class[_]

Definition Classes: AnyRef → Any
Annotations: @native()

def hashCode(): Int

Definition Classes: AnyRef → Any
Annotations: @native()

final def isInstanceOf[T0]: Boolean

Definition Classes: Any

def javaTopTopicsPerDocument(k: Int): JavaRDD[(Long, Array[Int], Array[Double])]

Java-friendly version of topTopicsPerDocument

Annotations: @Since( "1.5.0" )

lazy val javaTopicAssignments: JavaRDD[(Long, Array[Int], Array[Int])]

Java-friendly version of topicAssignments

Annotations: @Since( "1.5.0" )

def javaTopicDistributions: JavaPairRDD[Long, Vector]

Java-friendly version of topicDistributions

Annotations: @Since( "1.4.1" )

val k: Int

Number of topics

Definition Classes: DistributedLDAModel → LDAModel
Annotations: @Since( "1.3.0" )

lazy val logLikelihood: Double

Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)

Note:

This excludes the prior; for that, use logPrior.
Even with logPrior, this is NOT the same as the data log likelihood given the hyperparameters.

Annotations: @Since( "1.3.0" )

lazy val logPrior: Double

Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)

Annotations: @Since( "1.3.0" )

final def ne(arg0: AnyRef): Boolean

Definition Classes: AnyRef

final def notify(): Unit

Definition Classes: AnyRef
Annotations: @native()

final def notifyAll(): Unit

Definition Classes: AnyRef
Annotations: @native()

def save(sc: SparkContext, path: String): Unit

Save this model to the given path.

This saves:

human-readable (JSON) model metadata to path/metadata/
Parquet formatted data to path/data/

The model may be loaded using Loader.load.

sc: Spark context used to save model data.
path: Path specifying the directory in which to save this model. If the directory already exists, this method throws an exception.

Definition Classes: DistributedLDAModel → Saveable
Annotations: @Since( "1.5.0" )

final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes: AnyRef

def toLocal: LocalLDAModel

Convert model to a local model.

Convert model to a local model. The local model stores the inferred topics but not the topic distributions for training documents.

Annotations: @Since( "1.3.0" )

def toString(): String

Definition Classes: AnyRef → Any

def topDocumentsPerTopic(maxDocumentsPerTopic: Int): Array[(Array[Long], Array[Double])]

Return the top documents for each topic

maxDocumentsPerTopic: Maximum number of documents to collect for each topic.
returns: Array over topics. Each element represent as a pair of matching arrays: (IDs for the documents, weights of the topic in these documents). For each topic, documents are sorted in order of decreasing topic weights.

Annotations: @Since( "1.5.0" )

def topTopicsPerDocument(k: Int): RDD[(Long, Array[Int], Array[Double])]

For each document, return the top k weighted topics for that document and their weights.

returns: RDD of (doc ID, topic indices, topic weights)

Annotations: @Since( "1.5.0" )

lazy val topicAssignments: RDD[(Long, Array[Int], Array[Int])]

Return the top topic for each (doc, term) pair.

Return the top topic for each (doc, term) pair. I.e., for each document, what is the most likely topic generating each term?

returns: RDD of (doc ID, assignment of top topic index for each term), where the assignment is specified via a pair of zippable arrays (term indices, topic indices). Note that terms will be omitted if not present in the document.

Annotations: @Since( "1.5.0" )

val topicConcentration: Double

Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.

This is the parameter to a symmetric Dirichlet distribution.

Definition Classes: DistributedLDAModel → LDAModel
Annotations: @Since( "1.5.0" )
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.

def topicDistributions: RDD[(Long, Vector)]

For each document in the training set, return the distribution over topics for that document ("theta_doc").

returns: RDD of (document ID, topic distribution) pairs

Annotations: @Since( "1.3.0" )

lazy val topicsMatrix: Matrix

Inferred topics, where each topic is represented by a distribution over terms.

Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.

WARNING: This matrix is collected from an RDD. Beware memory usage when vocabSize, k are large.

Definition Classes: DistributedLDAModel → LDAModel
Annotations: @Since( "1.3.0" )

val vocabSize: Int

Vocabulary size (number of terms or terms in the vocabulary)

Definition Classes: DistributedLDAModel → LDAModel
Annotations: @Since( "1.3.0" )

final def wait(): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long, arg1: Int): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long): Unit

Definition Classes: AnyRef
Annotations: @native() @throws( ... )

Packages

DistributedLDAModel

Companion object DistributedLDAModel

class DistributedLDAModel extends LDAModel

Value Members

Inherited from LDAModel

Inherited from Saveable

Inherited from AnyRef

Inherited from Any

Ungrouped