Log In Sign Up

Topic Detection from Conversational Dialogue Corpus with Parallel Dirichlet Allocation Model and Elbow Method

by   Haider Khalid, et al.

A conversational system needs to know how to switch between topics to continue the conversation for a more extended period. For this topic detection from dialogue corpus has become an important task for a conversation and accurate prediction of conversation topics is important for creating coherent and engaging dialogue systems. In this paper, we proposed a topic detection approach with Parallel Latent Dirichlet Allocation (PLDA) Model by clustering a vocabulary of known similar words based on TF-IDF scores and Bag of Words (BOW) technique. In the experiment, we use K-mean clustering with Elbow Method for interpretation and validation of consistency within-cluster analysis to select the optimal number of clusters. We evaluate our approach by comparing it with traditional LDA and clustering technique. The experimental results show that combining PLDA with Elbow method selects the optimal number of clusters and refine the topics for the conversation.


Variable Selection for Latent Dirichlet Allocation

In latent Dirichlet allocation (LDA), topics are multinomial distributio...

Measuring LDA Topic Stability from Clusters of Replicated Runs

Background: Unstructured and textual data is increasing rapidly and Late...

Proactive Human-Machine Conversation with Explicit Conversation Goals

Though great progress has been made for human-machine conversation, curr...

Prediction, Selection, and Generation: Exploration of Knowledge-Driven Conversation System

In open-domain conversational systems, it is important but challenging t...

Contextual Topic Modeling For Dialog Systems

Accurate prediction of conversation topics can be a valuable signal for ...

Topic Discovery in Massive Text Corpora Based on Min-Hashing

The task of discovering topics in text corpora has been dominated by Lat...

Consistent Dialogue Generation with Self-supervised Feature Learning

Generating responses that are consistent with the dialogue context is on...