Contract Statements Knowledge Service for Chatbots

10/10/2019 ∙ by Boris Ruf, et al. ∙ AXA 0

Towards conversational agents that are capable of handling more complex questions on contractual conditions, formalizing contract statements in a machine readable way is crucial. However, constructing a formal model which captures the full scope of a contract proves difficult due to the overall complexity its set of rules represent. Instead, this paper presents a top-down approach to the problem. After identifying the most relevant contract statements, we model their underlying rules in a novel knowledge engineering method. A user-friendly tool we developed for this purpose allows to do so easily and at scale. Then, we expose the statements as service so they can get smoothly integrated in any chatbot framework.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

For a long time, researchers in artificial intelligence (AI) have been intrigued by the idea of developing a conversational agent that is capable of having a coherent conversation with humans 

[16, 5, 13]. Recent breakthroughs in semantics and speech recognition have given rise to hopes for robust solutions to the problem [10, 1]. Major information technology companies have released digital assistants and chatbot frameworks to facilitate the building of conversational agents [9, 7].

However, research studies on the user perception and expectations from users of chatbots indicate that the systems still require significant improvements in order to provide a meaningful experience [17]. Also, demand analysis identified a need for specialized digital assistants in customer facing processes, in particular in the insurance sector [8]. A promising approach to advance in this field are comprehensive knowledge engineering methodologies which back the chatbot and upgrade its conversational capabilities from small talk to domain expert [4].

Fig. 1: A sequence diagram of a sample conversation flow. The knowledge service incorporates the contract statement.

In our case, we are interested in the domain of contract statements of insurance policies. In order to narrow down the problem, we focus on the kind of questions that can be answered based on linear rules which take a finite set of parameter-value pairs as input. We assume that most customer support inquiries fall in the domain of our interest, i.e. questions around coverage and pricing in real-world insurance policies.

Modeling contracts bottom-up as self-executing contracts, also known as “smart contracts”, is appealing due to its large potential of automation and its resilience to tampering [2]. However, it is extremely hard to apply this concept on real-world contracts based on existing contract law provisions because smart contracts are “indifferent to the fundamental legal principles, such as lawfulness, fairness, and protection of the weaker party” [14].

Matching inquiries around a contract using a conventional FAQ bot would be an option, in particular when using a deep learning neural model for answer selection 

[6]. However, the use of static answers can quickly become complex and hard to maintain due to the arbitrary number of combinations.

In this paper, we propose a knowledge engineering methodology to model contract statements which support the automated handling of such questions. We choose a top-down approach, manually selecting the most relevant statements of a contract and modeling their underlying rules in a straightforward and accessible manner. The rules are based on combinations of different parameter spaces. This allows for the modelling of complex statements while facilitating easy maintainability. Conversational agents can query the rules via API which allows for seamless integration with any chatbot framework. We also assess the theoretical statement expressivity of our approach and demonstrate that it holds a significant potential of complexity reduction compared with conventional FAQ bots.

The remainder of this paper proceeds as follows. We begin with an overview of the overall architecture we propose and introduce a toy scenario. Next, in Section III, we define the expected input to the system, describe the theory behind our concept of contract statement modeling, and outline the different possible query scenarios. Then, Section IV studies the theoretical complexity reduction of our approach with respect to the maximum and the actual number of questions covered. Further, we provide the technical specification of our prototype which realises the present concept in Section V. Finally, we conclude by providing insights in possible future fields of research.

Fig. 2: Toy scenario: The hypergraph represents a contract statement of a fictional accident insurance statement.

Ii General architecture

The proposed architecture to respond to customer support inquiries consists of a chatbot client, a chatbot framework and several knowledge services. The conversation flow between those components has been illustrated in a sequence diagram in Figure 1.

Ii-a Chatbot client

The chatbot client is the messaging interface provided to interact with the customer support. It allows to send and receive short messages and preserves the conversation in a local message log.

Ii-B Chatbot framework

The chatbot framework manages the conversation flow and preprocesses the inquiries. In this paper, we focus on designing the knowledge service, the operations of the chatbot framework are not in the scope of this paper. Thus, we only briefly describe them as follows. Every inquiry sent to the chatbot framework is preprocessed using Natural Language Processing (NLP) methods. First, a text classifier is used for intent classification. By understanding the matter of the inquiry, the framework can identify the corresponding knowledge service. Then, named-entity recognition is run on the user message to retrieve the parameter-value pairs. Finally, the parameter-value pairs are dispatched to the knowledge service as query parameters.

Ii-C Knowledge service

Every knowledge service represents an endpoint for one specific type of question. It contains expert knowledge formalized as rules of a contract statement. The properties of the knowledge service are described in full length in Section III.

Ii-D Toy scenario

Throughout this paper we will illustrate all aspects of our proposal with a simple toy scenario. Its model is shown in Figure 2. The subject is a fictional accident insurance which covers sports accidents in different geographical regions. Prices vary depending on the kind of sport and the country. We would like to automate responses to price requests based on the customer’s parameters.

Fig. 3: The schematic representation of a hypergraph which models a contract statement. The vertices represent the different sets of parameter values and also the possible responses. Rules are defined by hyperedges which connect several vertices.

Iii Knowledge service

The knowledge service plays the role of the domain expert. It processes the query, identifies the matching rule and delivers the appropriate answer. In the following, we describe the expected input to the service, the modeling of a contract statement, and the different query scenarios.

Iii-a Input query

The knowledge service expects a structured request containing a set of query parameters as input. As previously described, this request is prepared by the chatbot framework.

In the context of our toy scenario, the customer could for example enter the question “How much is an accident insurance for scuba diving in Turkey?” in the chatbot client. The chatbot framework classifies the intent of the message and match the corresponding knowledge service which handles pricing questions. Further, it identifies the required parameters for this service, here sport and country. Using named-entity recognition, the relevant parameter values, in this example “scuba diving” and “Turkey”, are extracted and dispatched as query parameters to the knowledge service.

Iii-B Contract statement

We first propose to select relevant statements from a contract, that is, specific rules which are frequently subject of customer inquiries. We suggest to model such statements as hypergraphs. The vertices of a hypergraph represent the different parameter values and also the possible responses of each statement. Rules are defined by edges which connect several vertices.

In our toy scenario, the statement is about pricing. The vertices represent the possible parameter values, here groups of sports and countries. They also stand for the responses, in this example the different prices. The rules define the prices for each available combination of parameter values.

We choose a hypergraph to model a statement because, unlike a simple graph, a hypergraph has edges which can join any number of vertices. Let = be a finite, undirected hypergraph with vertices . is a set of hyperedges , such that each hyperedge contains an arbitrary set of vertices, . More precisely, is a subset of , where is the power set of .

An schematic presentation of a hypergraph as defined above can be found in Figure 3. An instance of this hypergraph representing the statement of the toy scenario is shown in Figure 2.

Iii-B1 Vertices

The vertices are split in 2 different types: the ones that represent the parameter values, and the ones that stand for the available responses.

Let vertices be the first subset of . They are each labeled with a string denoted as parameters where . is the set of strings which consist of at least 2 characters. In our toy scenario, the parameters are sport and country.

The labels are unique, , and the maximum number of parameters is the total number of vertices, thus . Each vertex of represents a set of possible values for the given parameter. We describe those values as set of keywords which consist of at least one string, thus where . The keywords are required to be unique per parameter, thus for each where Otherwise, several rules could apply simultaneously, and there would be no unambiguous result. In our continuous example, the keywords are , , etc.

Let vertices be the second subset of , labeled with responses where . Responses are to be unique in our model, thus . In the toy scenario, the responses are .

Iii-B2 Edges

Edges represent the statement’s rules. We require the rules to take at most one value per parameter, which is why we only allow edges to connect vertices of different parameters. Formally, this means that each hyperedge must contain vertices and so that . Also, it must include a minimum of 2 vertices, including 1 vertex of , so that .

For example, in our toy scenario displayed in Figure 2, the rules show that the prime for in would be cheaper than for in . On the other hand, insurance for in is not offered.

  Input: where
  Output:
  if  valid then
      Identify all vertices which are labeled with and include in the associated set of keywords
      Identify edge
     if  exists then
        
        
        
     else
        if  then
           
           
        else
           
           
        end if
     end if
  else
     
     
  end if
Algorithm 1 Outline rule matching algorithm

Iii-C Query scenarios

The necessary steps to find the right response to an inquiry have been outlined in the form of pseudo code in Algorithm 1. First, we identify all vertices of which match the parameter-value pairs provided in the query. Then, we search for a hyperedge that connects those vertices. For this purpose, we consider the following scenarios.

Iii-C1 Complete input query (default)

If parameter values were provided for all parameters, and a fully connected hyperedge exists, we single out the vertex which belongs to the subset . Notice that by definition each hyperedge must include exactly one such vertex. As response we send the label of this vertex to the chatbot framework.

If the matched vertices are not fully connected by a hyperedge, it means that none of the defined rules in the statement apply. Thus, the response is the boolean value . Depending on the use case, this response may be interpreted as “no”, or as “undefined” – impossible to make a valid statement.

Iii-C2 Incomplete input query

If the query is incomplete, a fully connected hyperedge may still exist, since we do not require hyperedges to include vertices of every single parameter. In this case, we send the label of the vertex which belongs to the subset , just as described above.

However, if the matched vertices are not represented by a fully connected hyperedge, the knowledge service prompts the user for the missing parameter values until either a hyperedge is found or the query is complete and contains values for all parameters. More specifically, the label of the first parameter without value is sent to the chatbot framework with HTTP status code 422 (“Unprocessable Entity”). The framework requests the parameter value from the user, and finally resubmits the updated parameter query. This approach enhances the knowledge service with a very basic capacity to manage part of the conversational flow.

For example, if the question “How much is an accident insurance for scuba diving?” was asked in the toy scenario, the knowledge service would respond with “country” and HTTP code 422, prior to providing a final response. The chatbot framework could rephrase the request to “In which country would you like to exercise the sport?”, include the answer of the user to the query and pass the completed parameter-value pair to the knowledge service.

Iii-C3 Invalid input query

A parameter query may include unexpected input, i.e. multiple values for the same parameter. For the sake of simplicity, the knowledge service currently only detects such exceptions and leaves it to the chatbot framework to handle them appropriately. More concretely, the knowledge service responds with the boolean value and the HTTP status code 400 (“Bad Request”).

Fig. 4: Statement expressivity scenarios for a differing number of vertices per parameter (#v) over an increasing number of keywords per vertex (#k). The number of parameters and the number of response vertices are fixed to 2 and 3 respectively for better comparability.
(a) Interactive hypergraph: The vertices are displayed as boxes. The 2 top layers contain the ones which represent parameter values, the bottom layer contains the response vertices. Edges of different color constitute the different rules.
(b) Keywords: For each vertex which represents a set of parameter values, matching keywords can be entered in the format of tags.
Fig. 7: Graphical user interface of the prototype

Iv Complexity reduction

In theory, the previously presented type of questions could be answered by conventional FAQ bots, too. However, such frameworks would require to write one explicit answer for every single question. This would result in a very large set of question-answer pairs, which is time-consuming to achieve and difficult to maintain. On the contrary, the contract model presented in this paper induces that each statement represented by a hypergraph factorizes all possible answers into rules and as such serves as dynamic answer to multiple questions.

Iv-a Statement expressivity

To estimate the maximum number of different questions that could be covered by a hypergraph, we need to add up all possible parameter-value pair combinations for any possible hyperedge. To achieve this, we first count the keywords represented by each vertex of

labeled with the same parameter and assign those numbers to a set . Then, we can compute with the following formula the value of which represents the maximum number of conventional, static question-answer pairs the present statement replaces:

(1)

where is the power set of , and is the element of each subset of .

We observe that the size of , which corresponds to the number of values accepted for one parameter vertex , as well as the total number of parameter layers , have a strong influence on the overall complexity.

For the toy scenario, the value of is calculated as follows:

As result, up to 34 conventional question-answer pairs can be mapped to this single hypergraph with 3 levels and 9 nodes.

Iv-B Total questions covered by a specific statement

In order to calculate the number of standard FAQ bot questions covered by a single given hypergraph, we sum up all possible parameter-value pair combinations for each hyperedge present in the graph. To achieve this, we first identify all vertices included in a hyperedge. Then, we count the keywords represented by each of those vertices and multiply the results. In order to compute , the total number of questions covered by the hypergraph, we repeat this procedure for all remaining hyperedges, and add up the sums. The resulting formula is:

(2)

where is the set of hyperedges, and .

For example, in the case of the toy scenario illustrated in Figure 2, the value of is computed as follows:

Hence, the statement modelled in the toy scenario substitutes 13 conventional FAQ bot questions.

Iv-C Impact of large number of keywords

It is worth to note that with we only estimate the upper bound, and , the number of actual total questions covered, may be considerably lower. However, in real-world scenarios the number of keywords (#k) can easily reach much higher values than illustrated in the toy scenario, for example when the keywords represent enumerations of cities or countries.

In Figure 4, we display varying statement expressivity scenarios. Concretely, we plot the maximum number of questions that could be covered for 3 different configurations. This value, , is computed as defined in Equation 1, with respect to the number of keywords per vertex (#k). In order to investigate the impact of one variable on , we keep the number of parameters fixed to 2, and the number of response vertices fixed to 3. In the 3 different scenarios, we set the number of vertices per parameter (#v) to 2, 3 and 4.

In general, we observe that the potential number of questions which can be covered by one statement grows exponentially over the number of keywords per vertex (#k) at high rates. Even for the very low values of #v, and chosen for the scenarios, rapidly reaches the four-digit range. Further, we see that a higher number of vertices per parameter (#v) increases the growth rate significantly.

V Implementation

Our prototype111Source code available at
https://github.com/axa-rev/reasoning-api-framework
is a multi-tenant web application implemented in Ruby on Rails, a web framework built on top of the object-oriented scripting language Ruby [12].

The graphical user interface is backed by the Bootstrap UI framework, and enhanced with the JavaScript diagramming library mxGraph [3, 11]. Figure (a)a shows the interactive user interface which is used to design a contract statement in the form of a hypergraph with vertices and edges. The vertex labels have been implemented as stacked layers. The 2 top layers contain the parameter values, the layer on the bottom the response vertices. Hyperedges can be created by simply drawing edges between the vertices. Different edge colors represent different hyperedges. The set of keywords associated with a vertex can be easily updated as displayed in Figure (b)b.

Implementing the quality engineering principle of poka-yoke, the edge validation built into the user interface guarantees hypergraph validity by design [15]. When drawing the edges, visual feedback helps the user to avoid logical errors and enforces the connection rules as defined in Section III-B. The different cases considered are displayed in Figure 12.

For seamless integration in any conversational agent, the web application exposes an Application Programming Interface (API). As response format we chose the open-standard, light-weight JavaScript Object Notation (JSON). The parameter-value pairs are included as query string in the API url.

Vi Conclusion

We presented a top-down knowledge engineering approach which helps improve the capabilities of conversational agents. As opposed to the static answers of conventional FAQ bots, our approach enables smarter, dynamic responses. The user-friendly graphical user interface allows for rapid contract statement creation and updating with no technical skills required. The significant reduction of complexity cuts the cost to manage the contract statements, which represent the essential knowledge necessary to provide meaningful responses.

A further plus of the presented architecture is that the statement rules are exposed through a generic API. This renders the system framework-agnostic and makes it possible to integrate with any chatbot framework. Also, updates of the statement come into effect instantly. Finally, isolating the knowledge and the input validation from the chatbot framework keeps the latter light and lean.

In the future, we want to extend the keyword matching to more sophisticated methods: Fuzzy matching could render the matching process more robust towards typos. Making use of word embeddings could match synonyms which are not explicitly included in the keyword sets. Further, the data types of the keyword sets should be expanded to ranges of numbers and dates. Also, we see potential to refine the conversational flow by ranking the missing parameter values by probability and plausibility. Eventually, a language-agnostic architecture will be investigated.

(a) Correct
(b) Wrong: Self-reference
(c) Wrong: Connection within same layer
(d) Wrong: Rule already exists
Fig. 12: The prototype features a built-in edge validation for error prevention.

References

  • [1] D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, E. Elsen, J. Engel, L. Fan, C. Fougner, T. Han, A. Y. Hannun, B. Jun, P. LeGresley, L. Lin, S. Narang, A. Y. Ng, S. Ozair, R. Prenger, J. Raiman, S. Satheesh, D. Seetapun, S. Sengupta, Y. Wang, Z. Wang, C. Wang, B. Xiao, D. Yogatama, J. Zhan, and Z. Zhu (2015) Deep speech 2: end-to-end speech recognition in english and mandarin. CoRR abs/1512.02595. External Links: Link, 1512.02595 Cited by: §I.
  • [2] M. Bartoletti and L. Pompianu (2017) An Empirical Analysis of Smart Contracts: Platforms, Applications, and Design Patterns BT - Financial Cryptography and Data Security. In Financial Cryptography and Data Security, M. Brenner, K. Rohloff, J. Bonneau, A. Miller, P. Y. A. Ryan, V. Teague, A. Bracciali, M. Sala, F. Pintore, and M. Jakobsson (Eds.), Cham, pp. 494–509. External Links: ISBN 978-3-319-70278-0 Cited by: §I.
  • [3] Bootstrap UI Framework. External Links: Link Cited by: §V.
  • [4] G. Cameron, D. Cameron, G. Megaw, R. Bond, M. Mulvenna, S. O. Neill, C. Armour, and M. Mctear (2018) Back to the Future : Lessons from Knowledge Engineering Methodologies for Chatbot Design and Development. Proceedings of British HCI 2018, pp. 1–5. Cited by: §I.
  • [5] K. M. Colby, S. Weber, and F. D. Hilf (1971) Artificial Paranoia. Artificial Intelligence 2 (1), pp. 1–25. External Links: Document, ISBN 0080181619, ISSN 00043702 Cited by: §I.
  • [6] J. Huang, T. Lin, and W. Lee (2018-10) Using deep learning and an external knowledge base to develop human-robot dialogues. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, pp. 3709–3714. External Links: Document Cited by: §I.
  • [7] S. Janarthanam (2017) Hands-On Chatbots and Conversational UI Development: Build Chatbots and Voice User Interfaces with Chatfuel, Dialogflow, Microsoft Bot Framework, Twilio, and Alexa Skills. Packt Publishing. External Links: ISBN 1788294661, 9781788294669 Cited by: §I.
  • [8] F. Koetter, M. Blohm, M. Kochanowski, J. Goetzer, D. Graziotin, and S. Wagner (2018) Motivations, classification and model trial of conversational agents for insurance companies. CoRR abs/1812.07339. External Links: Link, 1812.07339 Cited by: §I.
  • [9] G. López, L. Quesada, and L. A. Guerrero (2018) Alexa vs. siri vs. cortana vs. google assistant: a comparison of speech-based natural user interfaces bt - advances in human factors and systems interaction. In Alexa vs. Siri vs. Cortana vs. Google Assistant: A Comparison of Speech-Based Natural User Interfaces BT - Advances in Human Factors and Systems Interaction, I. L. Nunes (Ed.), Cham, pp. 241–250. External Links: ISBN 978-3-319-60366-7 Cited by: §I.
  • [10] G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur, D. Yu, and G. Zweig (2015)

    Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding

    .
    IEEE/ACM Transactions on Audio, Speech, and Language Processing 23 (3), pp. 530–539. External Links: Document, arXiv:1406.2661v1, ISBN 9781139058452, ISSN 2329-9290, Link Cited by: §I.
  • [11] mxGraph Diagramming Library. External Links: Link Cited by: §V.
  • [12] Ruby on Rails Web Framework. External Links: Link Cited by: §V.
  • [13] R. S. Wallace (2009-01) The anatomy of a.l.i.c.e. In Parsing the Turing Test, pp. 181–210. External Links: ISBN 978-1-4020-6710-5 Cited by: §I.
  • [14] A. Savelyev (2017-05) Contract law 2.0: ‘Smart’ contracts as the beginning of the end of classic contract law. Information & Communications Technology Law 26 (2), pp. 116–134. Note: doi: 10.1080/13600834.2017.1301036 External Links: Document, ISSN 1360-0834, Link Cited by: §I.
  • [15] S. Shingo (1986) Zero quality control: source inspection and the poka-yoke system. CRC Press. Cited by: §V.
  • [16] J. Weizenbaum (1966) ELIZA. Commun. ACM 9 (1). External Links: Document, ISSN 0001-0782 Cited by: §I.
  • [17] J. Zamora (2017) I’m sorry, dave, i’m afraid i can’t do that: chatbot perception and expectations. In Proceedings of the 5th International Conference on Human Agent Interaction, HAI ’17, New York, NY, USA, pp. 253–260. External Links: ISBN 978-1-4503-5113-3, Link, Document Cited by: §I.