Ranking ideas for diversity and quality

09/07/2017
by   Faez Ahmed, et al.
0

When selecting ideas or trying to find inspiration, designers often must sift through hundreds or thousands of ideas. This paper provides an algorithm to rank design ideas such that the ranked list simultaneously maximizes the quality and diversity of recommended designs. To do so, we first define and compare two diversity measures using Determinantal Point Processes (DPP) and additive sub-modular functions. We show that DPPs are more suitable for items expressed as text and that a greedy algorithm diversifies rankings with both theoretical guarantees and empirical performance on what is otherwise an NP-Hard problem. To produce such rankings, this paper contributes a novel way to extend quality and diversity metrics from sets to permutations of ranked lists. These rank metrics open up the use of multi-objective optimization to describe trade-offs between diversity and quality in ranked lists. We use such trade-off fronts to help designers select rankings using indifference curves. However, we also show that rankings on trade-off front share a number of top-ranked items; this means reviewing items (for a given depth like the top 10) from across the entire diversity-to-quality front incurs only a marginal increase in the number of designs considered. While the proposed techniques are general purpose enough to be used across domains, we demonstrate concrete performance on selecting items in an online design community (OpenIDEO), where our approach reduces the time required to review diverse, high-quality ideas from around 25 hours to 90 minutes. This makes evaluation of crowd-generated ideas tractable for a single designer. Our code is publicly accessible for further research.

READ FULL TEXT

page 16

page 18

research
10/27/2020

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics

The way pages are ranked in search results influences whether the users ...
research
04/29/2018

On Obtaining Stable Rankings

We often have to rank items with multiple attributes in a dataset. A typ...
research
04/15/2018

A Weighted Generalization of the Graham-Diaconis Inequality for Ranked List Similarity

The Graham-Diaconis inequality shows the equivalence between two well-kn...
research
11/13/2014

DUM: Diversity-Weighted Utility Maximization for Recommendations

The need for diversification of recommendation lists manifests in a numb...
research
12/10/2018

Top-N-Rank: A Scalable List-wise Ranking Method for Recommender Systems

We propose Top-N-Rank, a novel family of list-wise Learning-to-Rank mode...
research
01/15/2021

Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation

Crowdsourcing can collect many diverse ideas by prompting ideators indiv...

Please sign up or login with your details

Forgot password? Click here to reset