An Interpretable Deep Learning System for Automatically Scoring Request for Proposals

08/05/2020
by   Subhadip Maji, et al.
43

The Managed Care system within Medicaid (US Healthcare) uses Request For Proposals (RFP) to award contracts for various healthcare and related services. RFP responses are very detailed documents (hundreds of pages) submitted by competing organisations to win contracts. Subject matter expertise and domain knowledge play an important role in preparing RFP responses along with analysis of historical submissions. Automated analysis of these responses through Natural Language Processing (NLP) systems can reduce time and effort needed to explore historical responses, and assisting in writing better responses. Our work draws parallels between scoring RFPs and essay scoring models, while highlighting new challenges and the need for interpretability. Typical scoring models focus on word level impacts to grade essays and other short write-ups. We propose a novel Bi-LSTM based regression model, and provide deeper insight into phrases which latently impact scoring of responses. We contend the merits of our proposed methodology using extensive quantitative experiments. We also qualitatively asses the impact of important phrases using human evaluators. Finally, we introduce a novel problem statement that can be used to further improve the state of the art in NLP based automatic scoring systems.

READ FULL TEXT
research
06/01/2023

Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions

Automated scoring of student responses to open-ended questions, includin...
research
10/13/2021

Automated Essay Scoring Using Transformer Models

Automated essay scoring (AES) is gaining increasing attention in the edu...
research
02/28/2023

H-AES: Towards Automated Essay Scoring for Hindi

The use of Natural Language Processing (NLP) for Automated Essay Scoring...
research
08/06/2019

Word Embedding for Response-To-Text Assessment of Evidence

Manually grading the Response to Text Assessment (RTA) is labor intensiv...
research
02/23/2019

Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses

Automatically evaluating the quality of dialogue responses for unstructu...
research
11/17/2021

Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees

Automated Scoring (AS), the natural language processing task of scoring ...
research
01/05/2022

Automated Scoring of Graphical Open-Ended Responses Using Artificial Neural Networks

Automated scoring of free drawings or images as responses has yet to be ...

Please sign up or login with your details

Forgot password? Click here to reset