Calculating the similarity between words and sentences using a lexical database and corpus statistics

by   Atish Pawar, et al.

Calculating the semantic similarity between sentences is a long dealt problem in the area of natural language processing. The semantic analysis field has a crucial role to play in the research related to the text analytics. The semantic similarity differs as the domain of operation differs. In this paper, we present a methodology which deals with this issue by incorporating semantic similarity and corpus statistics. To calculate the semantic similarity between words and sentences, the proposed method follows an edge-based approach using a lexical database. The methodology can be applied in a variety of domains. The methodology has been tested on both benchmark standards and mean human similarity dataset. When tested on these two datasets, it gives highest correlation value for both word and sentence similarity outperforming other similar models. For word similarity, we obtained Pearson correlation coefficient of 0.8753 and for sentence similarity, the correlation obtained is 0.8794.



There are no comments yet.


page 3


A novel hybrid methodology of measuring sentence similarity

The problem of measuring sentence similarity is an essential issue in th...

Rethinking Crowd Sourcing for Semantic Similarity

Estimation of semantic similarity is crucial for a variety of natural la...

A Case Study to Reveal if an Area of Interest has a Trend in Ongoing Tweets Using Word and Sentence Embeddings

In the field of Natural Language Processing, information extraction from...

Similarity between Learning Outcomes from Course Objectives using Semantic Analysis, Blooms taxonomy and Corpus statistics

The course description provided by instructors is an essential piece of ...

Implanting Rational Knowledge into Distributed Representation at Morpheme Level

Previously, researchers paid no attention to the creation of unambiguous...

Modelling Creativity: Identifying Key Components through a Corpus-Based Approach

Creativity is a complex, multi-faceted concept encompassing a variety of...

Domain Specific Complex Sentence (DCSC) Semantic Similarity Dataset

Semantic textual similarity is one of the open research challenges in th...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.