Automated Query Expansion using High Dimensional Clustering
share
The exponential growth of information on the Internet has created a big challenge for retrieval systems in terms of yielding relevant results. This challenge requires automatic approaches for reformatting or expanding users' queries to increase recall. Query expansion (QE), a technique for broadening users' queries by appending additional tokens or phrases bases on semantic similarity metrics, plays a crucial role in overcoming this challenge. However, such a procedure increases computational complexity and may lead to unwanted noise in information retrieval. This paper attempts to push the state of the art of QE by developing an automated technique using high dimensional clustering of word vectors to create effective expansions with reduced noise. We implemented a command line tool, named Xu, and evaluated its performance against a dataset of news articles, concluding that on average, expansions generated using this technique outperform those generated by previous approaches, and the base user query.
READ FULL TEXT