Skip to content

Latest commit

 

History

History
17 lines (10 loc) · 1023 Bytes

spark-mllib-latent-dirichlet-allocation.adoc

File metadata and controls

17 lines (10 loc) · 1023 Bytes
Raw

Latent Dirichlet Allocation (LDA)

Note
 Information here are based almost exclusively from the blog post Topic modeling with LDA: MLlib meets GraphX.

Topic modeling is a type of model that can be very useful in identifying hidden thematic structure in documents. Broadly speaking, it aims to find structure within an unstructured collection of documents. Once the structure is "discovered", you may answer questions like:

  • What is document X about?

  • How similar are documents X and Y?

  • If I am interested in topic Z, which documents should I read first?

Spark MLlib offers out-of-the-box support for Latent Dirichlet Allocation (LDA) which is the first MLlib algorithm built upon GraphX.

Topic models automatically infer the topics discussed in a collection of documents.

Example

Caution
 FIXME Use Tokenizer, StopWordsRemover, CountVectorizer, and finally LDA in a pipeline.