Code and Data for ACL 2019 "Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention"
Semantically controlled neural response generation on limited-domain has achieved great performance. However, moving towards multi-domain large-scale scenarios is shown to be difficult because the possible combinations of semantic inputs grow exponentially with the number of domains. To alleviate such scalability issue, we exploit the structure of dialog acts to build a multi-layer hierarchical graph, where each act is represented as a root-to-leaf route on the graph. Then, we incorporate such graph structure prior as an inductive bias to build a hierarchical disentangled self-attention network, where we disentangle attention heads to model designated nodes on the dialog act graph. By activating different (disentangled) heads at each layer, combinatorially many dialog act semantics can be modeled to control the neural response generation. On the large-scale Multi-Domain-WOZ dataset, our algorithm can yield an improvement of over 5.0 BLEU score, and in human evaluation, it also significantly outperforms other baselines over various metrics including consistency, etc.READ FULL TEXT VIEW PDF
Dialogue act recognition is a fundamental task for an intelligent dialog...
Neural generative models have achieved promising performance on dialog
As a crucial component in task-oriented dialog systems, the Natural Lang...
Learning interpretable dialog structure from human-human dialogs yields ...
Open Domain dialog system evaluation is one of the most important challe...
The capacity of empathy is crucial to the success of open-domain dialog
Currently, open-domain generative dialog systems have attracted consider...
Code and Data for ACL 2019 "Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention"
Conversational artificial intelligenceYoung et al. (2013)
is one of the critical milestones in artificial intelligence. Recently, there have been increasing interests in industrial companies to build task-oriented conversational agentsWen et al. (2017); Li et al. (2017); Rojas-Barahona et al. (2017) to solve pre-defined tasks such as restaurant or flight bookings, etc (see Figure 1 for an example dialog from MultiWOZ Budzianowski et al. (2018)). Traditional agents are built based on slot-filling techniques, which requires significant human handcraft efforts. And it is hard to generate naturally sounding utterances in a generalizable and scalable manner. Therefore, different semantically controlled neural language generation models have been developed Wen et al. (2015, 2016a, 2016b); Dusek and Jurcícek (2016) to replace the traditional systems, where an explicit semantic representation (dialog act) are used to influence the RNN generation. The canonical approach is proposed in Wen et al. (2015)2016b), these models though achieving good performance on limited domains, suffer from scalability problem as the possible dialog acts grow combinatorially with the number of domains.
In order to alleviate such issue, we propose a hierarchical graph representation by leveraging the structural property of dialog acts. Specifically, we first build a multi-layer tree to represent the entire dialog act space based on their inter-relationships. Then, we merge the tree nodes with the same semantic meaning to construct an acyclic multi-layered graph, where each dialog act is interpreted as a root-to-leaf route on the graph. Such graph representation of dialog acts not only grasps the inter-relationships between different acts but also reduces the exponential representation cost to almost linear, which will also endow it with greater generalization ability. Instead of simply feeding such vectorized representation as an external feature vector to the neural networks, we propose to incorporate such a structure act as an inductive prior for designing the neural architecture, which we name as hierarchical disentangled self-attention network (HDSA). InFigure 2
, we show how the dialog act graph structure is explicitly encoded into model architecture. Specifically, HDSA consists of multiple layers of disentangled self-attention modules (DSA). Each DSA has multiple switches to set the on/off state for its heads, and each head is bound for modeling a designated node in the dialog act graph. At the training stage, conditioned on the given dialog acts and the target output sentences, we only activate the heads in HDSA corresponding to the given acts (i.e., the path in the graph) to activate the heads with their designated semantics. At test time, we first predict the dialog acts and then use them to activate the corresponding heads to generate the output sequence, thereby controlling the semantics of the generated responses without handcrafting rules. As depicted inFigure 2, by gradually activating nodes from domain action slot, the model is able to narrow its response down to specifically querying the user about the color and type of the taxi, which provides both strong controllability and interpretability.
Experiment results on the large-scale MultiWOZ dataset Budzianowski et al. (2018) show that our HDSA significantly outperforms other competing algorithms.111The code and data are released in https://github.com/wenhuchen/HDSA-Dialog In particular, the proposed hierarchical dialog act representation effectively improves the generalization ability on the unseen test cases and decreases the sample complexity on seen cases. In summary, our contributions include: (i) we propose a hierarchical graph representation of dialog acts to exploit their inter-relationships, which greatly reduces the sample complexity and improves generalization, (ii) we propose to incorporate the structure prior in semantic space to design HDSA to explicitly model the semantics of neural generation, and outperforms baselines.
Canonical task-oriented dialog systems are built as pipelines of separately trained modules: (i) user intention classification Shi et al. (2016); Goo et al. (2018), which is for understanding human intention. (ii) belief state tracker Williams et al. (2013); Mrksic et al. (2017a, b); Zhong et al. (2018); Chen et al. (2018), which is used to track user’s query constraint and formulate DB query to retrieve entries from a large database. (iii) dialog act prediction Wen et al. (2017)
, which is applied to classify the system action. (iv) response generationRojas-Barahona et al. (2017); Wen et al. (2016b); Li et al. (2017); Lei et al. (2018) to realize language surface form given the semantic constraint. In order to handle the massive number of entities in the response, Rojas-Barahona et al. (2017); Wen et al. (2016b, 2015) suggest to break response generation into two steps: first generate delexicalized sentences with placeholders like Res.Name, and then post-process the sentence by replacing the placeholders with the DB record. The existing modularized neural models have achieved promising performance on limited-domain datasets like DSTC Williams et al. (2016), CamRes767 Rojas-Barahona et al. (2017) and KVRET Eric et al. (2017), etc. However, a recently introduced multi-domain and large-scale dataset MultiWOZ Budzianowski et al. (2018) poses great challenges to these approaches due to the large number of slots and complex ontology. Dealing with such a large semantic space remains a challenging research problem.
, and then decompose it into two parts: the lower part (blue rectangle) contains state tracking and symbolic DB execution, the upper part consists of dialog act prediction and response generation conditioned on the state tracking and DB results. In this paper, we are particularly interested in the upper part (act prediction and response generation) by assuming the ground truth belief state and DB records are available. More specifically, we set out to investigate how to handle the large semantic space of dialog acts and leverage it to control the neural response generation. Our approach encodes the history utterances into distributed representations to predict dialog acts and then uses the predicted dialog acts to control neural response generation. The key idea of our model is to devise a more compact structured representation of the dialog acts to reduce the exponential growth issue and then incorporate the structural prior for the semantic space into the neural architecture design. Our proposed HDSA is inspired by the linguistically-inform self-attentionStrubell et al. (2018), which combines multi-head self-attention with multi-task NLP tasks to enhance the linguistic awareness of the model. In contrast, our model disentangles different heads to model different semantic conditions in a single task, which provides both better controllability and interpretability.
Dialog acts are defined as the semantic condition of the language sequence, comprising of domains, actions, slots, and values.
The dialog acts have universally hierarchical property, which is inherently due to the different semantic granularity. Each dialog act can be seen as a root-to-leaf path as depicted in Figure 4222we add dummy node “none” to transform those non-leaf acts into leaf act to normalize all acts into triplet; for example “hotel-inform” is converted into “hotel-inform-none”. Such tree structure can capture the kinship between dialog acts, i.e. “restaurant-inform-location” has stronger similarity with “restaurant-inform-name” than “hotel-request-address”. The canonical approach to encode dialog acts is by concatenating the one-hot representation at each tree level into a flat vector like SC-LSTM Wen et al. (2015); Budzianowski et al. (2018) (details are in in Github333https://github.com/andy194673/nlg-sclstm-multiwoz/blob/master/resource/woz3/template.txt). However, such representation impedes the cross-domain transfer between different slots and the cross-slot transfer between different values (e.g the “recommend” under restaurant domain is different from “recommend” under hospital domain). As a result, the sample complexity can grow combinatorially as the potential dialog act space expands in large-scale real-life dialog systems, where the potential domains and actions can grow dramatically. To address such issue, we propose a more compact graph representation.
The tree-based representation cannot capture the cross-branch relationship like “restaurant-inform-location” vs. “hotel-inform-location”, leading to a huge expansion of the tree. Therefore, we propose to merge the cross-branch nodes that share the same semantics to build a compact acyclic graph in the right part of Figure 4444We call it graph because now one child node can have multiple parents, which violates the tree’s definition.. Formally, we let denote the set of all the original dialog acts. And for each act , we use to denote its -layer graph form, where is its one-hot representation in the layer of the graph. For example, a dialog act “hotel-inform-name” has a compact graph representation . More formally, let denote the number of nodes at the layer of , respectively. Ideally, the total representation cost can be dramatically decreased from tree-based representation to = in our graph representation. Due to the page limit, we include the full dialog act graph and its corresponding semantics in the Appendix. When multiple dialog acts are involved in the single response, we propose to aggregate them as as the -dimensional graph representation, where denotes the bit-wise OR operator555For example, two acts, and , are aggregated into ..
Compared to the tree-based representation, the proposed graph representation under strong cross-branch overlap can greatly lower the sample complexity. Hence, it leads to great advantage under sparse training instances. For example, suppose the exact dialog act “hotel-recommend-area” never appears in the training set. Then, at test time when used for response generation, the flat representation will obviously fail. In contrast, with our hierarchical representation, “hotel”, “recommend” and “area” may have appeared separately in other instances (e.g., “recommend” appears in “attraction-recommend-name”). Its graph representation could still be well-behaved and generalize well to the unseen (or less frequent) cases due to the strong compositionality.
Figure 5 gives an overview of our dialog system. We now proceed to discuss its components below.
We first explain the utterance encoder module, which uses a neural network to encode the dialog history (i.e., concatenation of previous utterances from both the user and the system turns ), into distributed token-wise representations with its overall representation as follows:
where can be CNN, LSTM, Transformer, etc, are the representation. The overall feature is used to predict the hierarchical representation of dialog act. That is, we output a vector , whose
component gives the probability of thenode in the dialog act graph being activated:
where is the attention matrix, the weights are the learnable parameters to project the input to space, and
is the Sigmoid function. Here, we followBudzianowski et al. (2018); Rojas-Barahona et al. (2017) to use one-hot vector and for representing the DB records and belief state (see the original papers for details). For convenience, we use to collect all the parameters of the utterance encoder and action predictor. At training time, we propose to maximize the cross-entropy objective as follows:
where denotes the inner product between two vectors. At test time, we predict the dialog acts , where is the threshold and is the indicator function.
Recently, the self-attention-based Transformer model has achieved state-of-the-art performance on various NLP tasks such as machine translation Vaswani et al. (2017), and language understanding Devlin et al. (2018); Radford et al. (2018). The success of the Transformer is partly attributed to the multi-view representation using multi-head attention architecture. Unlike the standard transformer which concatenates vectors from different heads into one vector, we propose to uses a switch to activate certain heads and only pass through their information to the next level (depicted in the right of Figure 5). Hence, we are able to disentangle the attention heads to model different semantic functionalities, and we refer to such module as the disentangled self-attention (DSA). Formally, we follow the canonical Transformer Vaswani et al. (2017) to define the Scaled Dot-Product Attention function given the input query/key/value features as:
where denotes the sequence length of the input, denotes query, key and value. Here, we use different self attention functions with their independent parameterization to compute the multi-head representation as follows:
where the input matrices are computed from the input token embedding , and denotes the dimension of the embedding. The head adopts its own parameters , , to compute the output . We shrink the dimension at each head to to reduce the computation cost as suggested in Vaswani et al. (2017).
We first use the cross-attention network to incorporate the encoded dialog history
, and then we apply a position-wise feed forward neural network, a layer normalization , and a linear projection layer to obtain . These layers are shared across different heads. The main innovation of our architecture lies in disentangling the heads. That is, instead of concatenating to obtain the layer output like the standard Transformer, we employ a binary switch vector to control different heads and aggregate them as a output matrix . Specifically, the -th row of , denoted as , can be understood as the output corresponding to the -th input token in the response. This approach is similar to a gating function to selectively pass desired information. By manipulating the attention-head switch , we can better control the information flow inside the self-attention module. We illustrate the gated summation over multi-heads in Figure 6.
When the dialog system involves more complex ontology, the semantic space can grow rapidly. In consequence, a single-layer disentangled self-attention with a large number of heads is difficult to handle the complexity. Therefore, we further propose to stack multiple DSA layers to better model the huge semantic space with strong compositionality. As depicted in Figure 3, the lower layers are responsible for grasping coarse-level semantics and the upper layers are responsible for capturing fine-level semantics. Such progressive generation bears a strong similarity with human brains in constructing precise responses. In each DSA layer, we feed the utterance encoding and last layer output as the input to obtain the newer output matrix . We collect the output from the last DSA layer to compute the joint probability over a observed sequence , which can be decomposed as a series of product over the probabilities:666We follow the standard approach in Transformer to use a mask to make depend only on during training. And during test time, we decode sequentially from left-to-right.
where and are the projection weight and bias onto a vocabulary of size , is the index, denotes the softmax operation, denotes the set of the attention switches , , over the layers, and denotes all the decoder parameters.
Recall that the graph structure of dialog acts is explicitly encoded into HDSA as a prior, where each head in HDSA is set to model a designated semantic node on the graph. In consequence, the hierarchical representation can be used to control the head switch . At training time, the model parameters are optimized from the training data triple to maximize the likelihood of ground truth acts and responses given the dialog history. Formally, we propose to maximize the following objective function as follows:
At test time, we propose to use the predicted dialog act to control the language generation. The errors can be seen as coming from two sources, one is from inaccurate dialog act prediction, the other is from imperfect response generation.
To evaluate our proposed methods, we use the recently proposed MultiWOZ dataset Budzianowski et al. (2018) as the benchmark, which was specifically designed to cover the challenging multi-domain and large-scale dialog managements (see the summary in Table 1). This new benchmark involves a much larger dialog action space due to the inclusion of multiple domains and complex database backend. We represent the 625 potential dialog acts into a three-layered hierarchical graph that with a total nodes (see Appendix for the complete graph).
|Dialogs||Total Turns||Unique Tokens||Value|
We follow Budzianowski et al. (2018) to select 1000 dialogs as the test set and 1000 dialogs as the development set. And we mainly focus on the context-to-response problem, with the dialog act prediction being a preliminary task. The best HDSA uses three DSA layers with 10/7/27 heads to separately model the semantics of domain, actions and slot (dummy head is included to model “none” node). Adam Kingma and Ba (2014) with a learning rate of is used to optimize the objective. A beam size of 2 is adopted to search the hypothesis space during decoding with vocabulary size of 3,130. Also, by small-scale search, we fix the threshold due to better empirical results.
|None||LSTM Budzianowski et al. (2018)||18.8||71.2||60.2||54.8||15.1|
|3-layer Transformer Vaswani et al. (2017)||19.1||71.1||59.9||55.1||15.2|
|Tree Act||SC-LSTM Wen et al. (2015)||20.5||74.5||62.5||57.7||16.6|
|Graph Act (Predicted)||3-layer Transformer-out||22.5||80.8||64.8||64.2||19.3|
|Straight DSA (44 heads) + 2 x SA||22.6||80.3||67.1||65.0||20.0|
|2-layer HDSA (7/27 heads) + SA||23.2||82.9||69.1||65.1||20.3|
|3-layer HDSA (10/7/27 heads)||23.6||82.9||68.9||65.7||20.6|
|Graph Act (Groundtruth)||3-layer Transformer-in||29.1||85.5||72.6||83.8||25.1|
|Straight DSA (44 heads) + 2 x SA||29.6||86.4||75.6||84.1||25.5|
|3-layer HDSA (10/7/27 heads)||30.4||87.9||78.0||86.2||26.2|
We first train dialog act predictors using different neural networks to compare their performances. The experimental results (measured in F1 scores) are reported in Table 2. Experimental results show that fine-tuning the pre-trained BERT Devlin et al. (2018) can lead to significantly better performance than the other models. Therefore, we will use it as the dialog act prediction model in the following experiments. Instead of jointly training the predictor and the response generator, we simply fix the trained predictor when learning the generator .
We follow Budzianowski et al. (2018) to use delexicalized-BLEU Papineni et al. (2002), inform rate and request success as three basic metrics to compare the delexicalized generation against the delexicalized reference. We further propose Entity F1 Rojas-Barahona et al. (2017) to evaluate the entity coverage accuracy (including all slot values, days, numbers, and reference, etc), and restore-BLEU to compare the restored generation against the raw reference. The evaluation metrics are detailed in the supplementary material.
Before diving into the experiments, we first list all the models we experiment with as follows:
Without Dialog Act, we use the official code 777https://github.com/budzianowski/multiwoz: (i) LSTM Budzianowski et al. (2018): it uses history as the attention context and applies belief state and KB results as side inputs. (ii) Transformer Vaswani et al. (2017): it uses stacked Transformer architecture with dialog history as source attention context.
With Sparse Tree Dialog Act, we feed the tree-based representation as an external vectors into different architectures. (i) SC-LSTM Wen et al. (2015): it feeds the sparse dialog act to the semantic gates to control the generation process. (ii) Transformer-in: it appends the sparse dialog act vector to input word embedding (iii) Transformer-out: it appends the sparse dialog act vector to the last layer output, before the softmax function.
With Compact Graph Dialog Act (Predicted), we use the proposed graph representation for dialog acts and use it to control the natural language generation. (i) Transformer-in/out: it uses the flattened graph representation and feeds it as an external embedding feature. (ii) Straight DSA: it uses the flattened graph representation and model it with a one-layer DSA followed with two layers of self-attention. (iii) 2-layer HDSA: it adopts the partial action/slot levels of hierarchical graph representation, used as an ablation study. (iv) 3-layer HDSA: it adopts the full 3-layered hierarchical graph representation, used for the main model.
With Graph Dialog Act (Groundtruth): it uses the ground truth dialog acts as input to see the performance upper bound of the proposed response generator architecture.
In order to make these models comparable, we design different hidden dimensions to make their total parameter size comparable. We demonstrate the performance of different models in Table 3, and briefly conclude with the following points: (i) by feeding the sparse tree representation to input/output layer (Transformer-in/out), the model is not able to capture the large semantics space of dialog acts with sparse training instances, which unsurprisingly leads to restricted performance gain against without dialog act input. (ii) the graph dialog act is essential in reducing the sample complexity, the replacement can lead to significant and consistent improvements across different models. (iii) the hierarchical graph structure prior is an efficient inductive bias; the structure-aware HDSA can better model the compositional semantic space of dialog acts to yield a decent gain over Transformer-in/out with flattened input vector. (vi) our approaches yield significant gain (10+%) on the Inform/Request success rate, which reflects that the explicit structured representation of dialog act is very effective in guiding dialog response in accomplishing the desired tasks. (v) the generator is greatly hindered by the predictor accuracy, by feeding the ground truth acts, the proposed HDSA is able to achieve an additional gain of 7.0 in BLEU and 21% in Entity F1.
To better understand the performance gain of the hierarchical graph-based representation, we design synthetic tests to examine its generalization ability. Specifically, we divide the dialog acts into five categories based on their frequency of appearance in the training data: very few shot (1-100 times), few shot (100-500 times), medium shot (500-2K times), many shot (2K-5K times), and very many shot (5K+ times). We compute the average BLEU score of the turns within each frequency category and plot the result in Figure 7. First, by comparing Transformer-in with compact Graph-Act against Transformer-in with sparse Tree-Act, we observe that for few number shots, the graph act significantly boosts the performance, which reflects our conjecture to lower sample complexity and generalize better to unseen (or less frequent) cases. Furthermore, by comparing Graph-Act Transformer-in with HDSA, we observe that HDSA ahieves better results by exploiting the hierarchical structure in dialog act space.
Owing to the low consistency between automatic metrics and human perception on conversational tasks, we also recruit trustful judges from Amazon Mechanical Turk (AMT) (with prior approval rate 95%)888https://www.mturk.com/ to perform human comparison between the generated responses from HDSA and SC-LSTM. Three criteria are adopted: (i) relevance: the response correctly answers the recent user query. (ii) coherence: the response is coherent with the dialog history. (iii) consistency: the generated sentence is semantically aligned with ground truth. During the evaluation, each AMT worker is presented two responses separately generated from HDSA and SC-LSTM, as well the ground truth dialog history. Each HIT assignment has 5 comparison problems, and we have a total of 200 HIT assignments to distribute. In the end, we perform statistical analysis on the harvested results after rejecting the failure cases and display the statistics in Table 4.
From the results, we can observe that our model significantly outperforms SC-LSTM in the coherence, i.e., our model can better control the generation to maintain its coherence with the dialog history.
In order to quantitatively compare the controllability of HDSA, Graph-Act Tranformer-in, and SC-LSTM, we further design a synthetic NLG experiment, where we randomly pick 50 dialog history as the context from test set, and then randomly select 3 dialog acts and their combinations as the semantic condition to control the model’s responses generation. We demonstrate an example in the supplementary to visualize the evaluation procedure. Quantitatively, we hire human workers to rate (measured in match, partially match, and totally mismatch) whether the model follows the given semantic condition to generate coherent sentences. The experimental results are reported in the bottom half of Table 4, which demonstrate that both the compact dialog act representation and the hierarchical structure prior are essential for controllability.
The proposed graph representation works well under the cases where the set of domain slot-value pairs have significant overlaps, like Restaurant, Hotel, where the knowledge is easy to transfer. Under occasions where such exact overlap is scarce, we propose to use group similar concepts together as hypernym and use one switch to control the hypernym, which can generalize the proposed method to the broader domain.
A trade-off that we found in our structure-based encoding scheme is that: when multiple dialog acts are involved with overlaps in the action layer, ambiguity will happen under the graph representation. For example, the two dialog acts “restaurant-inform-price” and “hotel-inform-location” are merged as “[restaurant, hotel] [inform] [price, location]”, the current compressed representation is unable to distinguish them with “hotel-inform-price” or “restaurant-inform-location”. Though these unnatural cases are very rare in the given dataset without hurting the performance per se, we plan to address such pending expressiveness problem in the future research.
In this paper, we propose a new semantically-controlled neural generation framework to resolve the scalability and generalization problem of existing models. Currently, our proposed method only considers the supervised setting where we have annotated dialog acts, and we have not investigated the situation where such annotation is not available. In the future, we intend to infer the dialog acts from the annotated responses and use such noisy data to guide the response generation.
We really appreciate the efforts of the anonymous reviews and cherish their valuable comments, they have helped us improve the paper a lot. We are gratefully supported by a Tencent AI Lab Rhino-Bird Gift Fund. We are also very thankful for the public available dialog dataset released by University of Cambridge and PolyAI.
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 5016–5026.
Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 3732–3741.
Here we detailedly explain the model implementation of the baselines and our proposed HDSA model. In the encoder side, we use a three-layered transformer with input embedding size of 64 and 4 heads, the dimension of query/value/key are all set to 16, in the output layer, the results of 4 heads are concatenated to obtain a 64-dimensional vector, which is the first broadcast into 256-dimension and then back-projected to 64-dimension. By stacking three layers of such architecture, we obtain at the end the series of 64-dimensional vectors. Following BERT, we use the first symbol as the sentence-wise representation , and compute its matching score against all the tree node to predict the representation of dialog acts .
In the decoder, we adopt take as input any length features , each with dimension of 64, in the first layer, since we have 10 heads, the dimension for each head is 6, thus the key, query feature dimensions are fixed to 6, the second layer with dimension of 9, the third with dimension of 2. The value feature is all fixed to 16, which is equivalent to the encoder side. After self-attention, the position-wise feed-forward neural network projects each feature back to 64 dimensions, which is further projected to 3.1K vocabulary dimension to model word probability.
We simply demonstrate an example of our automatic evaluation metrics in Figure 9.
Here we visualize how we feed the dialog act input in as an embedding into the transformer to control the sequence generation process as Figure 8.
To better understand the human evaluation procedure, we demonstrate the user interface in Figure 10.
To better understand the results, we depict an example in Figure 11, where 3 different dialog acts are picked as the semantic condition to constrain the response generation.
Here we first enumerate the node semantics of the graph representation as follows:
Domain-Layer 10 choices: ’restaurant’, ’hotel’, ’attraction’, ’train’, ’taxi’, ’hospital’, ’police’, ’bus’, ’booking’, ’general’.
Action-Layer 7 choices: ’inform’, ’request’, ’recommend’, ’book’, ’select’, ’sorry’, ’none’.
Slot-Layer 27 choices: ’pricerange’, ’id’, ’address’, ’postcode’, ’type’, ’food’, ’phone’, ’name’, ’area’, ’choice’, ’price’, ’time’, ’reference’, ’none’, ’parking’, ’stars’, ’internet’, ’day’, ’arriveby’, ’departure’, ’destination’, ’leaveat’, ’duration’, ’trainid’, ’people’, ’department’, ’stay’.
Then we enumerate the entire graph as follows: