Log In Sign Up

Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing

by   Akshat Shrivastava, et al.

Task-oriented semantic parsing models have achieved strong results in recent years, but unfortunately do not strike an appealing balance between model size, runtime latency, and cross-domain generalizability. We tackle this problem by introducing scenario-based semantic parsing: a variant of the original task which first requires disambiguating an utterance's "scenario" (an intent-slot template with variable leaf spans) before generating its frame, complete with ontology and utterance tokens. This formulation enables us to isolate coarse-grained and fine-grained aspects of the task, each of which we solve with off-the-shelf neural modules, also optimizing for the axes outlined above. Concretely, we create a Retrieve-and-Fill (RAF) architecture comprised of (1) a retrieval module which ranks the best scenario given an utterance and (2) a filling module which imputes spans into the scenario to create the frame. Our model is modular, differentiable, interpretable, and allows us to garner extra supervision from scenarios. RAF achieves strong results in high-resource, low-resource, and multilingual settings, outperforming recent approaches by wide margins despite, using base pre-trained encoders, small sequence lengths, and parallel decoding.


page 1

page 2

page 3

page 4


Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing

An effective recipe for building seq2seq, non-autoregressive, task-orien...

Low-Resource Task-Oriented Semantic Parsing via Intrinsic Modeling

Task-oriented semantic parsing models typically have high resource requi...

Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling

As an essential task in task-oriented dialog systems, slot filling requi...

Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing

A common recent approach to semantic parsing augments sequence-to-sequen...

MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

Scaling semantic parsing models for task-oriented dialog systems to new ...

X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing

Task-oriented compositional semantic parsing (TCSP) handles complex nest...

Diagnosing Transformers in Task-Oriented Semantic Parsing

Modern task-oriented semantic parsing approaches typically use seq2seq t...

1 Introduction

Figure 1: High-Level Overview. Retrieve-and-Fill (RAF) consists of 4 steps: (1) Encode the utterance via an utterance encoder (e.g., RoBERTa); (2) Encode all scenarios Desai et al. (2021) in the scenario bank into a cached index via a scenario encoder (e.g., RoBERTa); (3) Compute the dot product similarity amongst the incoming utterance and the index of all scenarios, obtaining the top- candidates; (4) For each retrieved scenario, leverage the non-autoregressive span pointer decoder Shrivastava et al. (2021) to impute each scenario’s spans.

Task-oriented conversational assistants typically first use semantic parsers to map textual utterances into structured frames for language understanding Hemphill et al. (1990); Coucke et al. (2018); Gupta et al. (2018); Rongali et al. (2020); Aghajanyan et al. (2020). While these parsers achieve strong performance with rich supervision, they often face obstacles adapting to novel settings, especially ones with distinct semantics and scarce data. Recent approaches address this by improving parsers’ data efficiency, such as by using pre-trained representations Aghajanyan et al. (2020); Rongali et al. (2020)

, optimizing loss functions

Chen et al. (2020b), and supplying natural language prompts Desai et al. (2021)

. However, these approaches typically rely on larger models and longer contexts, impeding their applicability in real-world conversational assistants. Even though non-autoregressive models can alleviate some concerns

Babu et al. (2021); Shrivastava et al. (2021); Zhu et al. (2020), to the best of our knowledge, there exists no approach which strikes an appealing balance between model size, runtime latency, and cross-domain generalizability.

We begin tackling this problem by introducing scenario-based task-oriented semantic parsing which is more closely tied to how new task domains are developed. A slight variation on original semantic parsing, we are given access to all supported scenarios apriori and have to parse the utterance given this scenario bank. Here, a scenario is akin to a incomplete frame; it is, precisely, an intent-slot template with variables as leaf spans (e.g., IN:GET_WEATHER [SL:LOCATION ] ]), indicating it maps to a family of linguistically similar utterances. As domain development and data-collection usually starts out with designing the set of supported scenarios, our intuition is that by explicitly giving the model access to this bank we can train it to more explicitly reason about scenarios and improve performance especially in the low data regime.

Concretely, we propose RAF (Retrieve-and-Fill), a modular yet differentiable architecture for scenario-based task-oriented semantic parsing. Guided by the definition of our task, RAF proceeds in two steps: (1) given an utterance, a retrieval module finds the highest ranking scenario and (2) given the utterance and retrieved scenario, a filling module imputes spans into the scenario, creating the final frame. This approach requires no extra supervision despite performing auxiliary inference: utterances and frames are typically provided, and scenarios are obtained by stripping leaf text in frames. We design RAF to capitalize on the advantages of prior work but avoid their disadvantages; using base pre-trained encoders across-the-board, our retrieval module caches intrinsic representations during inference and our filling module non-autoregressively decodes leaf spans in a generalizable fashion.

We evaluate our approach in high-resource, low-resource, and multilingual settings using standard task-oriented semantic parsing datasets. RAF achieves 87.52% EM on TOPv2 Chen et al. (2020b) and 86.14% EM on TOP Gupta et al. (2018), outperforming recent autoregressive and non-autoregressive models. RAF also excels in weakly supervised settings: on TOPv2-DA Desai et al. (2021), we outperform Inventory Desai et al. (2021) on 4 domains (alarm, music, timer, weather) despite using <128 token sequence lengths and 2-4x less parameters, on TOPv2-LR Chen et al. (2020b), we outperform BART Lewis et al. (2020), RINE Mansimov and Zhang (2021), and RoBERTa + Span Pointer Shrivastava et al. (2021) by wide margins on weather, and on MTOP Li et al. (2021), we outperform XLM-R + Span Pointer Shrivastava et al. (2021), achieving 42% EM averaged across en{es, fr, de, hi, th} transfer tasks.

To summarize, our contributions are: (1) Introducing scenario-based task-oriented semantic parsing, a novel task which requires disambiguating scenarios during typical utteranceframe predictions; (2) Creating RAF (Retrieve-and-Fill), a modular yet differentiable architecture composed of a retrieval and filling module for solving scenario-based task-oriented semantic parsing, leveraging novel intrinsic representations of scenarios; and (3) Achieving strong results in high-resource, low-resource, and multilingual settings, outperforming recent models such as Span Pointer, Inventory, and RINE by large margins, while also optimizing for model size, runtime latency, and cross-domain generalizability.

Utterance Frame
what’s the weather in seattle [IN:GET_WEATHER [SL:LOCATION seattle ] ]
how’s the forecast in sf [IN:GET_WEATHER [SL:LOCATION sf ] ]
what’s the weather in seattle tomorrow [IN:GET_WEATHER [SL:LOCATION seattle ] [SL:DATE_TIME tomorrow ] ]
how’s the forecast in sf at 8pm [IN:GET_WEATHER [SL:LOCATION sf ] [SL:DATE_TIME 8pm ] ]
Table 1: Scenario Description. Scenarios are intent-slot templates with missing slot text, suggesting they subclass linguistically similar utterances. We show examples of utterances, scenarios, and frames in the weather domain; each scenario consists of multiple (utterance, frame) pairs.

2 Scenario-based Semantic Parsing

We formally introduce the task of scenario-based semantic parsing. Task-oriented conversational assistants support a wide range of domains (e.g., calling, reminders, weather) to maximize coverage over users’ needs. Over time, in response to requests and feedback, developers often iterate on assistants’ skill-sets by adding new domains. Though the crowdsourcing process of collecting and annotating samples—utterances and frames—for new domains can be accomplished in many ways, we propose a two-step methodology where developers (a) develop scenarios which roughly describe a family of samples and (b) collect linguistically-varied samples consistent with each scenario. We elaborate more on our scenario-based methodology below.

Scenario Definition.

We define a scenario as an intent-slot rule which abstracts away linguistic variation in utterances. More specifically, it is a decoupled semantic frame Aghajanyan et al. (2020) with variables in leaf spans, indicating it can subclass utterances with similar syntactic and semantic structure, an example is showing in table 1. Our notion of a scenario is inspired by production rules in constituency parsing Chomsky (1959), but the parallel is not exact given our scenarios have semantic not syntactic types.

Using scenarios, we can effectively quantize the space of possible utterances by identifying and defining slices of user requests. Our solution also offers fine-grained control over precision and recall, which is important in real-world systems; we can collect more paraphrases to improve precision and we can create more scenarios to improve recall.

Case Study: Weather.

Figure 1 shows an example setting where we crowdsource weather domain samples using the scenario-based methodology outlined above. To begin, we may envision building a weather system which supports requests with location and/or date-time information. Therefore, we can define two scenarios: (1) a family of samples with one location span; and (2) a family of samples with one location span and one date-time span. Each scenario explicitly outlines the intents and slots that must be present, as scenarios are not “one-size-fits-all” rules with optional slotting. So, as an example, the utterance “how’s the forecast in sf” would not be compatible with the scenario [IN:GET_WEATHER [SL:LOCATION ] [SL:DATE_TIME ] ] since it does not have a date-time span as specified by the variable.

Generalization Types.

There are multiple types of generalization developers may care about when building conversational systems; we specifically focus on two, namely intra-scenario and extra-scenario generalization. By intra-scenario generalization, we refer to a systems’ ability to parse linguistic variations of the same scenario, and by extra-scenario generalization, we refer to a systems’ ability to parse scenarios with unseen intent-slot combinations. Using the weather example in Table 1, a system trained on the first scenario only may exhibit the following behavior: with intra-scenario generalization, it would be able to parse utterances inside the training scenario, and with extra-scenario, it would be able to parse utterances outside the training scenario. Our focus is primarily on intra-scenario generalization, as we argue high precision over an illustrative set of scenarios is more important in the early stages of domain development. But, extra-scenario generalization is an interesting avenue of future work, as it offers opportunities to explore techniques with data augmentation Andreas (2020), inductive biases Oren et al. (2020), compositionality Gupta et al. (2018), etc.

Task Definition.

Finally, we precisely define our task of scenario-based semantic parsing. For the typical task of semantic parsing, we define

as a random variable over utterances,

as a random variable over frames, and model : find the most likely frame given the utterance. However, because we introduce scenarios as a coarse, intermediate representation of frames, we additionally define as the set of all supported scenarios given a priori, and model .

3 RAF: Retrieve and Fill

We propose a model called RAF (Retrieve and Fill) for our scenario-based semantic parsing task that naturally decomposes the task into a coarse-to-fine objective where we (a) find the most likely scenario given the utterance and (b) find the most likely frame given the utterance and scenario. More concretely given an utterance and scenarios , as well as a gold scenario and gold frame , we learn our model as follows:

  1. Retrieve (Coarse Step; §3.1): A retrieval module maximizes by learning to retrieve scenario given utterance , e.g., “what’s the weather in seattle” [IN:GET_WEATHER [SL:LOCATION ] ].

  2. Fill (Fine Step; §3.2): A filling module maximizes by decoding the most likely frame given the structure of scenario and spans of utterance , e.g., [IN:GET_WEATHER [SL:LOCATION ] ] [IN:GET_WEATHER [SL:LOCATION seattle ] ].

RAF is a composable yet differentiable model which implements coarse-to-fine processing: we first develop a coarse-grained sketch of an utterance’s frame, then impute fine-grained details to achieve the final frame. In these types of approaches, there often exists a trade-off when creating the intermediate representation; if it is “too coarse”, the filling module suffers, but if it is “too fine”, the retrieval module suffers. We find scenarios, as defined in §2, offer the most appealing solution, as the retrieval module unearths rough syntactic-semantic structure, while the filling module focuses in on imputing exact leaf spans.

This objective is similar, in spirit, to other setups in syntactic and semantic parsing Dong and Lapata (2018), but we explore different parameterizations of the likelihood functions. In the following sub-sections, we discuss the technical details behind the retrieval and filling modules, as well as describe the training and inference procedures.

3.1 Retrieval Module

First, we discuss the coarse-grained step of RAF, which aims to find the best-fitting scenario for an utterance.

A typical solution is learning a classifier which explicitly maximizes the conditional probability

of an utterance mapping to a scenario

. The classifier can be parameterized with a pre-trained encoder, and its parameters can be learned via maximum likelihood estimation. However, the multi-class classification approach has a couple of disadvantages; each scenario

contains useful signal in its structure (e.g., its intents and slots) which is not utilized and the output space itself cannot be “hot swapped” during inference, such as if new scenarios were to be dynamically added.

An alternative is formulating this task as a metric learning problem. Here, we can maximize the similarity between an utterance and scenario as judged by a scalar metric. This offers numerous advantages: we can explore ad-hoc encodings of utterances and scenarios, adjust the output space dynamically during inference, and compute exact conditional probabilities by leveraging a (tractable) partition function.

3.1.1 Bi-Encoder Retrieval

Following retrieval modeling in open-domain QA Karpukhin et al. (2020), we specifically leverage pre-trained encoders ( for utterances and

for scenarios) to compute dense vector representations, then maximize the dot product similarity

between utterance-scenario pairs ; the precise nature of is discussed in §3.1.3. To learn such a metric space and avoid degenerate solutions we need to train the encoders to pull positive pairs together and push negative pairs apart. Hence, we need access to both positive (gold; ) and negative (non-gold; ) pairs.

3.1.2 Negatives Sampling

The choice of negatives has a large impact on retrieval performance, which is consistent with findings in information retrieval Zhan et al. (2021); Karpukhin et al. (2020). We explore two types of negative sampling to improve retrieval performance: in-batch negatives and model-based negatives.

In-Batch Negatives.

We mine positive and negative pairs from each training batch using in-batch negatives Karpukhin et al. (2020). Let and be the utterance and scenario matrices, each being a matrix consisting of -dimensional embeddings up to batch size . We obtain similarity scores upon computing the similarity matrix , where consists of positive scores and consists of negative scores. For each positive utterance-scenario pair , we now have negative pairs . Having collected multiple negative pairs per positive pair, we leverage the contrastive loss defined in Karpukhin et al. (2020); Chen et al. (2020a):

Model-Based Negatives.

While in-batch negatives greatly expands the number of negatives, these negatives may not be particularly challenging as they are randomly sampled from the training dataset. To increase the quality of our utterance-scenario metric, we explore augmenting in-batch negatives with model-based negatives. Specifically, given a retrieval module from training round with metric , we find the top- neighbors for each positive pair by computing . Using this algorithm, we cache each training example’s hard negatives, then in training round , we fine-tune the retrieval module using both in-batch and model-based negatives. Each non-seed training round therefore features at most negatives per positive pair. This procedure is similar to iterative training used in Oğuz et al. (2021).

Identity Masking.

The precise number of negatives is empirically slightly less, as sometimes we see conflicts between each training examples’ negative pairs. Let and be two training examples within the same batch. During model-based negatives sampling in training round , may include as a top- negative pair. This complicates metric learning as becomes a positive and negative for simultaneously. We therefore implement an identity mask on each training example’s negatives which ensures no conflicts when mixing in-batch and model-based negatives.

3.1.3 Scenario Representation

While we have covered scenario-based retrieval above, we have not yet precisely described how dense vectors for scenarios are computed. Recall our definition of utterance-scenario similarity in §3.1.1: our objective is to maximize , where is a string transformation applied to scenarios.

Because we parameterize and with standard pre-trained encoders (e.g., RoBERTa), it follows that and must return approximately similar vectors in order to have a high dot product. However, a task setup using vanilla utterances and scenarios does not achieve this. Consider the utterance “what’s the weather in seattle” and scenario IN:GET_WEATHER [SL:LOCATION ] ]. Pre-trained encoders can return strong return representations for the utterance, but not necessarily for the scenario; because they have been pre-trained on naturally-occurring, large-scale text corpora, they are unlikely to glean as much value from dataset-specific concepts, such as IN:GET_WEATHER and SL:LOCATION.

An effective solution to this problem, as proposed by Desai et al. (2021), leverages intrinsic modeling to rewrite intents and slots as a composition of their intrinsic parts in a single string. Desai et al. (2021) define two, in particular: the categorical type (e.g., “intent” or “slot”) and language span (e.g., “get weather” or “location”). Using this methodology, we can transform the scenario IN:GET_WEATHER [SL:LOCATION ] ] [ intent | get weather [ slot | location ] ], which is inherently more natural and descriptive.

Guided by the general concept of intrinsic modeling, our goal here is to define such that , all else being equal. We discuss our approach in detail in the following sub-sections.

3.1.4 Language Spans

Desai et al. (2021) chiefly use an automatic method to extract language spans from ontology labels. For example, using standard string processing functions, we can extract “get weather” from IN:GET_WEATHER. While this method is a surprisingly strong baseline, it heavily relies on a third-party developers’ notion of ontology nomenclature, which may not always be pragmatically useful. In TOPv2 Chen et al. (2020b), our principal evaluation dataset, there exists ambiguous labels like SL:AGE—is this referring to the age of a person, place, or thing?

Therefore, to improve consistency and descriptiveness, we propose a handmade method where we manually design language spans for each ontology label.111See Appendix B for our curated intent and slot descriptions, respectively. This approach is similar to prompting Brown et al. (2020); Schick and Schütze (2021); Gao et al. (2020); Shin et al. (2020), but unlike typical prompt-engineering methods, we attempt to assign intuitive descriptions with no trial-and-error. While it is certainly possible to achieve better results with state-of-the-art methods, our aim is to be as generalizable as possible while still reaping the benefits of descriptive prompting.

The most frequent techniques we use are (1) using the label as-is (e.g., IN:GET_WEATHER “get weather”); (2) inserting or rearranging prepositions (e.g., IN:ADD_TO_PLAYLIST_MUSIC “add music to playlist”; and (3) elaborating using domain knowledge (e.g., IN:UNSUPPORTED_ALARM “unsupported alarm request”).

3.1.5 Example Priming

Despite using curation to improve ontology label descriptions, there are still many labels which remain ambiguous. One such example is SL:SOURCE; this could refer to a travel source or messaging source, but without seeing its exact manifestations, it is challenging to fully grasp its meaning. This motivates us to explore example priming: augmenting scenario representations with randomly sampled, dataset-specific slot values. This can help our model further narrow down the set of spans each slot maps to during parsing. Furthermore, our examples are just spans, so they are straightforward to incorporate into our representation. For example, for the slot SL:WEATHER_TEMPERATURE_UNIT, we can augment and contextualize its representation “slot | unit of weather temperature” with “slot | unit of weather temperature | F / C” where “F” and “C” are examples which appear in our dataset.

3.1.6 Representation Sampling

Our scenario-based retrieval task performs reasoning over a compact set of scenarios, unlike large-scale, open-domain tasks such as information retrieval and question answering which inculcate millions of documents. As such, the chance our system overfits to a particular representation is much greater, no matter what it is set to. So, we instead make stochastic by uniformly sampling unique frame representations for encoding scenarios. We re-define

as a discrete random variable with outcomes

where . Each denotes a unique scenario representation (e.g., a string using curated descriptions without examples); Table 11 enumerates the complete set of outcomes. While we intend to improve the generalizability of our system during training, we typically want deterministic behavior during testing, and therefore we restrict the outcomes to a singleton .

3.2 Filling Module

Next, we discuss the fine-grained step of RAF, which aims to infill a retrieved scenario with utterance spans in leaf slots. Unlike the coarse-grained step, the fine-grained step can be modeled with off-the-shelf models from prior work in task-oriented semantic parsing.

This step is typically modeled as a seq2seq problem where a decoder maps a scenario to a frame using cross-attention on an utterance , which results in the objective . Because frame is a sequence, this objective can be factorized in multiple ways: an autoregressive approach assumes conditional dependence on prior frame tokens while a non-autoregressive approach assumes conditional independence. We elect to use the latter given its recent successes in task-oriented semantic parsing; we refer interested readers to Babu et al. (2021); Shrivastava et al. (2021) for technical details. We largely use their model as-is, but one modification we make is scenario fusion. Rather than the decoder learning an embedding matrix for scenario tokens from scratch, we initialize the decoder’s embeddings with the scenario encoder’s final state. Our final objective is where refers to label smoothing Pereyra et al. (2017).

3.3 Training and Inference

Once we combine the coarse-grained and fine-grained steps, we now have an end-to-end system which maps an utterance (“get weather in seattle”) to a frame with ontology tokens (intents and slots) and utterance spans (e.g., [IN:GET_WEATHER [SL:LOCATION seattle ] ]).

Because our system requires strong negatives for accurate retrieval, we decompose training into three steps: (1) Train RAF using in-batch negatives only; (2) Sample the top- frames as model-based negatives from RAF in Step (1) by running it over the training set; (3) Train RAF using both in-batch negatives and model-based negatives. RAF requires three sets of parameters to train: the retrieval module consists of an utterance encoder and scenario encoder , while the filling module consists of a frame decoder . Our system is modularized and differentiable, therefore we fine-tune these modules using a joint loss where is a scalar weighting term. In the low-resource setting We further augment our loss with an optional R3F term Aghajanyan et al. (2021) to encourage robust representations following Shrivastava et al. (2021). Hyper parameter details are described in Appendix §C.

During inference, we can pipeline RAF’s components: given an utterance , the retrieval module finds the best scenario , and using the predicted scenario as a template, the filling module outputs a frame with ontology tokens (intents and slots) and utterance spans. Importantly, to maintain inference efficiency, the scenario encoders’ embeddings over the scenario set can be cached, since these are known a priori.

4 Experiments and Results

We evaluate RAF in three settings: a high-resource setting (100,000+ training samples), low-resource setting (1-1,000 training samples), and multilingual setting (0 training samples). Our goal here is to show that our system both achieves competitive performance on established benchmarks and offers substantial benefits in resource-constrained environments where training samples are limited.

4.1 Datasets for Evaluation

Following prior work in task-oriented semantic parsing Babu et al. (2021); Aghajanyan et al. (2020); Shrivastava et al. (2021); Mansimov and Zhang (2021); Desai and Aly (2021); Desai et al. (2021); Gupta et al. (2021); Rongali et al. (2020), we use 5 datasets for evaluation: TOP Gupta et al. (2018), TOPv2 Chen et al. (2020b), TOPv2-LR (Low Resource; Chen et al. (2020b)), TOPv2-DA (Domain Adaptation; Desai et al. (2021)), and MTOP Li et al. (2021). TOP and TOPv2 are used for high-resource experiments, TOPv2-LR and TOPv2-DA are used for low-resource experiments, and MTOP is used in multilingual experiments.

4.2 Systems for Comparison

We compare against multiple task-oriented semantic parsing models, which cover autoregressive (AR), and non-autoregressive (NAR) training. See Aghajanyan et al. (2020); Mansimov and Zhang (2021); Babu et al. (2021); Shrivastava et al. (2021) for detailed descriptions of these models.

The autoregressive models consist of BART Lewis et al. (2020) and RoBERTa Liu et al. (2019), and RINE Mansimov and Zhang (2021), and the non-autoregressive models are RoBERTa NAR Babu et al. (2021) and RoBERTa NAR + Span Pointer Shrivastava et al. (2021). These models are applicable to both high-resource and low-resource settings; though, for the latter, we also add baselines from Desai et al. (2021): CopyGen (BART + copy-gen decoder) and Inventory (BART + intrinsic modeling). The multilingual setting only requires swapping RoBERTa with XLM-R Conneau et al. (2020).

We denote our system as RAF in our experiments. Unless noted otherwise, we use RoBERTa for the utterance encoder and secnario and a random-init, copy-gen, transformer decoder for the frame decoder . As alluded to before, we swap RoBERTa with XLM-R for multilingual experiments.

Model TOPv2 TOP
Type: Autoregressive Modeling (Prior)
RoBERTa 86.62 83.17
RoBERTa 86.25 82.24
BART 86.73 84.33
BART 87.48 85.71
Type: Non-Autoregressive Modeling (Prior)
RoBERTa 85.78 82.37
 + Span Pointer 86.93 84.45
RoBERTa 86.25 83.40
 + Span Pointer 87.37 85.07
Type: Scenario Modeling (Ours)
RAF 87.14 86.00
RAF 87.52 86.14
Table 2: High-Resource Results. Exact Match (EM) on TOPv2 Chen et al. (2020b) and TOP Gupta et al. (2018). We compare various semantic parsing paradigms: autoregressive, non-autoregressive, and scenario. RAF achieves strong performance on TOPv2 and TOP, illustrating its competitiveness with state-of-the-art models.
alarm music timer weather
CopyGen 47.24 25.58 16.62 47.24
CopyGen 36.91 23.84 32.64 53.08
Inventory 62.13 23.00 28.92 54.53
Inventory 67.25 38.68 48.45 61.77
RAF (ours) 62.71 35.47 55.06 61.05
Table 3: High-Difficulty Low-Resource Results. EM on the 1 SPIS split of TOPv2-DA Desai et al. (2021). Compared to Inventory and CopyGen baselines, RAF achieves competitive performance with a fraction of parameter usage.
Weather Domain (SPIS)
10 25 50 100 500 1000
Type: Autoregressive Modeling (Prior)
RoBERTa AR 69.71 74.90 77.02 78.69 86.36
BART AR 73.34 73.35 76.58 79.16 86.25
RINE 74.53 87.80
RINE 77.03 87.50
Type: Non-Autoregressive Modeling (Prior)
RoBERTa NAR 59.01 72.12 73.41 78.48 87.42
 + Span Pointer 72.03 74.74 74.85 78.14 88.47
Type: Scenario Modeling (Ours)
RAF 75.10 78.74 77.53 79.67 87.91 88.17
Table 4: Medium-Difficulty Low-Resource Results. EM on various SPIS splits of the TOPv2 Chen et al. (2020b) weather domain. RAF largely outperforms autoregressive and non-autoregressive models, trailing RoBERTa-Base + Span Pointer only in a high-resource split.
Zero-Shot Evaluation
en enes enfr ende enhi enth Avg
XLM-R NAR 78.3 35.2 32.2 23.6 18.1 16.7 25.2
 + Span Pointer 83.0 51.2 51.4 42.0 29.6 27.3 40.3
RAF (ours) 81.1 56.0 54.9 40.1 32.1 30.0 42.6
Table 5: Multilingual Results. We perform zero-shot experiments where we fine-tune a parser on English (en), then evaluate it a non-English language—Spanish (es), French (fr), German (de), Hindi (hi), and Thai (th)—without fine-tuning. Average EM (Avg) is taken over the five non-English languages. RAF outperforms XLM-R-Base + Span Pointer by +2.3% on average.

4.3 High-Resource Setting

First, we evaluate RAF in a high-resource setting where hundreds of thousands are samples are available for supervised training; Table 2 shows the results. RAF achieves strong results across-the-board, using both base and large pre-trained encoders: RAF consistently outperforms other base variants by 0.25-0.5 EM and RAF comparatively achieves the best results on TOP and TOPv2.

4.4 Low-Resource Setting

Having established our system is competitive in high-resource settings, we now turn towards evaluating it in low-resource settings, where training samples are not as readily available. Here, we chiefly consider two setting types: a high difficulty setting (TOPv2-DA) with 1-10 samples and a medium difficulty setting (TOPv2-LR) with 100-1,000 samples. The exact number of samples in a few-shot training subset depend on both the subset’s cardinality and sampling algorithm.

Tables 3 and 4 show results on the high and medium difficulty settings, respectively. RAF achieves competitive results in the high-difficulty setting, outperforming both CopyGen and Inventory by large margins; notably, on timer, we nearly double Inventory’s exact match score. RAF also performs well in the medium-difficulty setting; our system consistently outperforms prior autoregressive, and non-autoregressive models. These results particularly highlight the effectiveness of coarse-to-fine modeling: our retrieval module learns a generalizable notion of utterancescenario mappings, especially given that utterance and scenarios are scored on semantic alignment, and our filling module precisely infills utterance spans into the scenario template.

4.5 Multilingual Setting

Finally, we consider a multilingual setting, where a model trained on English samples undergoes zero-shot transfer to non-English samples. In Table 5, we see that, compared to XLM-R NAR + Span Pointer, RAF achieves +2.3 EM averaged across all 5 non-English languages. Upon inspecting this result more closely, RAF’s performance is strong across both typologically similar languages (+4.8 EM on Spanish, +3.5 EM on French) and distinct languages (+2.7 EM on Hindi and Thai). A key reason for RAF’s strong performance is most of the domain shift is localized to the retrieval module. In MTOP, utterances across multiple languages have linguistic variation but their scenarios are the exact same, and so a bulk of our system’s work is in retrieving the correct scenario. While our results illustrate the efficacy of a monolingual retriever, we can explore creating a multilingual retriever in future work; due to our system’s modularity, such a retriever can simply be a drop-in replacement for the current one.

5 Ablations and Analysis

While our experiments show RAF achieves strong performance in high-resource, low-resource, and multilingual settings, we have not yet isolated the contribution of its different components. Here, we perform several experiments to better understand RAF’s design decisions.

5.1 Model Ablations

Model EM EM-S
Classify and Fill 84.80 87.16
RAF 87.03 89.34
 - Hard Negatives 83.69 86.00
 - Identity Masking 85.87 88.23
 - Scenario Fusion 86.26 88.69
 - Parameter sharing 86.76 89.10
 - Repr. Sampling 86.70 89.05

 + Heuristic negatives

85.66 89.10
Table 6: Model Ablations. We assess several components of our model, individually removing them and evaluating EM (Exact Match) and EM-S (Exact Match of Scenarios, i.e., intent-slot templates without slot text) on the TOPv2 Chen et al. (2020b) validation set.
Source Fine-Tuning Target Fine-Tuning
Canonical Repr. 86.67 88.74 74.25 76.80
Intrinsic Repr. 87.10 89.15 78.74 81.70
 - Automatic 87.02 89.06 78.92 81.80
 - Handmade 86.85 88.87 79.27 82.29
 - Examples 86.91 88.90 78.25 80.89
Table 7: Scenario representation ablation, where the target domain is a 25 SPIS split of the TOPv2 Chen et al. (2020b) weather domain. Following the typical few-shot fine-tuning methodology, we perform source fine-tuning on all domains except weather and reminder, then perform target fine-tuning on a specific split. Encouragingly, we see intrinsic representations result in better EM and EM-S.

First, we perform model ablations on RAF, removing core retrieval- and filling-related components we originally introduced in §3. From Table 6, we draw the following conclusions:

Negatives are important for accurate scenario retrieval.

The metric learning objective for retrieval, as introduced in §3.1.2, precisely delineates between positive and negative samples. Our ablations show model-based negatives and identity masking are critical to achieve best performance; when removing model-based negatives, for example, retrieval accuracy drops by 3%+. We also investigate training RAF with heuristic-based negatives: a simple algorithm which finds top- similar scenarios with string-based edit distance (§A). However, heuristic-based negatives regress both retrieval-only and end-to-end approach, suggesting model-based negatives are more informative.

Sharing parameters between retrieval encoders improves quality.

Our retrieval module has two encoders: an utterance encoder and a scenario encoder . An important design decision we make is tying both encoders’ parameters together with RoBERTa Liu et al. (2019); this improves end-to-end performance by roughly +0.6 EM. We believe that parameter sharing among retrieval encoders improves generalizability: because there are more vastly more unique utterances than scenarios, the scenario encoder may overfit to a select set of scenarios, so weight tying enables the joint optimization of both encoders.

Scenario fusion enables better end-to-end modeling.

Because RAF is composed of two neural modules—the retrieval and filling modules—chaining them together arbitrarily may result in information loss. The filling module chiefly uses scenario token embeddings to reason over the retrieval module’s outputs. Our results show that scenario fusion, initializing these embeddings using the scenario encoder’s final state, improves upon random init by +0.77 EM and +0.65 EM-S.

Weather Domain (SPIS)
0 10 25 50 100 500 1000 Avg
 Standard Retrieval + Standard Filling 26.19 75.10 78.74 77.53 79.67 87.91 88.17 73.33
 Oracle Retrieval + Standard Filling 81.68 90.43 92.13 91.97 93.35 95.74 96.27 91.65
 Standard Retrieval + Oracle Filling 27.67 77.84 81.70 79.92 81.78 89.88 90.44 75.60
Table 8: Evaluating whether retrieval or filling is the most challenging components of RAF in low-resource settings. We fine-tune several variants of RAF, using either a standard / oracle retriever and a standard / oracle filler, on various SPIS splits of the TOPv2 Chen et al. (2020b) weather domain. RAF with oracle retrieval achieves the best performance, suggesting utterancescenario retrieval is the most difficult piece to model.
Zero-Shot Evaluation
en enes enfr ende enhi enth Avg
 Standard Retrieval + Standard Filling 81.1 56.0 54.9 40.1 32.1 30.0 42.6
 Oracle Retrieval + Standard Filling 91.0 68.9 71.6 67.5 47.5 48.9 60.9
 Standard Retrieval + Oracle Filling 83.8 66.5 62.8 45.5 40.2 46.7 52.3
Table 9: Evaluating whether retrieval or filling is the most challenging components of RAF in low-resource settings. See Table 8 for a description of our methodology; we use MTOP Li et al. (2021) for evaluation instead.

5.2 Representation Ablations

Our results above demonstrate representation sampling is a key piece of end-to-end modeling. But, do we need to include all representations outlined in Appendix B in our sampling scheme? The principal reason we use intrinsic representations is for cross-domain robustness, and so we focus our ablation on a low-resource setting, Weather (25 SPIS) in TOPv2-LR. Using the typical low-resource fine-tuning algorithm Chen et al. (2020b), we adapt multiple versions of RAF to Weather, each omitting a representation (intrinsic automatic, handmade, or examples) from the sampling algorithm.

Table 7 shows these results. When comparing the high-level scenario representation, we see that canonical (e.g., “[IN:GET_WEATHER [SL:LOCATION ]”) underperforms intrinsic (e.g., “[ intent | get weather [slot | location ] ]”) by a wide margin (-4.49%) in the target domain. This implies RAF better understands scenarios’ natural language descriptions even if they contain unseen, domain-specific terms. Furthermore, we see leveraging all concepts (handmade, automatic, examples) achieves both competitive source and target performance. Even though excluding handmade improves target performance (+0.53%), it regresses source performance (-0.28%), suggesting sampling all representations is more generalizable.

5.3 Retrieval vs. Filling

We now turn towards better understanding the aspects our model struggles with. Because RAF jointly optimizes both the retrieval and filling modules, one question we pose is whether the retrieval or filling task is more difficult. We create three versions of RAF: (1) standard retrieval + standard filling, (2) oracle retrieval + standard filling, and (3) standard retrieval + oracle filling. By comparing models (1), (2), and (3), we can judge the relative difficulty of each task.

We begin by evaluating these models in a high-resource setting; on the TOPv2 eval dataset, the standard model gets 87.03%, retrieval oracle gets 96.56%, and filling oracle gets 89.34%. Here, the gap between models (1)-(2) is +9.53%, while the gap between models (1)-(3) is +2.31%, indicating the retrieval module is the main performance bottleneck. We also perform experiments in low-resource and multilingual settings, displaying results in Tables 8 and 9, respectively. These results also confirm the same trend: in both settings, the retrieval oracle achieves the best performance, notably achieving +18.3% averaged across 5 multilingual transfer experiments.

Despite retrieval having the most room for improvement, we also see some evidence filling struggles in certain multilingual transfer cases; for example, providing gold spans can improve enth transfer by +16.7%. As such, there is ample opportunity for optimizing the retrieval and filling modules in future work.

5.4 Extra-Scenario Generalization

A core difference between scenario-based and non-scenario-based (seq2seq; autoregressive or non-autoregressive) models is that scenario-based models “know” of all scenarios beforehand, while seq2seq models do not, and therefore have to purely rely on generalization. We further quantify the impact that this has by dividing overall EM using two groups: (1) Known vs. Unknown - scenarios in the training dataset vs. test dataset and (2) In-Domain vs. Out-of-Domain - scenarios with a supported intent vs. unsupported intent (e.g., IN:UNSUPPORTED_*).

From the results in Table 10, we draw a couple of conclusions. First, RAF outperforms on Unknown EM given that we index unique scenarios across the train, eval, and test datasets before fine-tuning. Second, RAF outperforms on In-Domain EM but underperforms on Out-of-Domain EM. Because RAF leverages intrinsic descriptions of scenarios, the word “unsupported” may not precisely capture what it means for an utterance to be in- vs. out-of-domain.

Model EM Known EM Unknown EM ID EM OOD EM
RAF 87.14 88.30 59.96 88.67 44.11
SpanPointer 86.76 88.50 46.20 88.07 49.70
BART 86.72 88.33 49.15 88.03 49.69
Table 10: Comparing EM on Known vs. Unknown and In-Domain (ID) vs. Out-of-Domain (OD) frames. RAF performs better on unknown frames, but struggles with out-of-domain frames.
Figure 2: Visualizing the semantic space for the weather domain as the model is trained on more and more data. We visualize the index prior to any training (0 SPIS) to low-resource (10, 25 SPIS) and high resource (1000 SPIS). The graphs are TSNE projections of the utterance vectors used for retrieval color coded by the scenario each utterance belongs to.
Figure 3: Depicting scenario visualizations, where each is a TSNE projection of utterances belonging to the specified scenario color coded by the predicted scenario. (A) covers the scenario [IN:CREATE_ALARM [SL:ALARM_NAME ] [SL:DATE_TIME ] showing a case of ambiguity of alarm names. (B) covers the scenario [IN:PLAY_MUSIC [SL:MUSIC_TYPE ] where annotations are missing the slot “music type”. (C) covers the scenario [IN:SET_DEFAULT_PROVIDER_MUSIC [SL:MUSIC_PROVIDER ] showing how our model requires more support here as the predictions for this scenario span OOD and music domain.

6 Visualizations

Because RAF encodes utterances and scenarios into a joint embedding space, we can directly visualize this space to further understand our models’ inner-workings. We present two case studies: domain development (§6.1) and error analysis (§6.2).

6.1 Domain Development

Figure 2 presents an example of performing domain development on the weather domain. Here, we train RAF with 4 dataset sizes (0 SPIS, 10 SPIS, 25 SPIS, and 1,000 SPIS) to simulate zero-shot, few-shot, low-resource, and high-resource settings, respectively. Each utterance (from the high-resource split) is projected using an utterance encoder and colored according to its gold scenario. Interestingly, the zero-shot setting has multiple, apparent clusters, but the overall performance is poor given many scenarios overlap with each other. The clusters spread further apart as the dataset size increases, suggesting the scenarios become more well-defined.

6.2 Error Analysis

While we have demonstrated how RAF refines the utterance-scenario space as we increase dataset size, we now dive deeper into how each space can be used to further analyze domain semantics. In figure 3, using our high-resource-trained RAF model, we create multiple scenario spaces: each utterance is projected using an utterance encoder and colored according to its predicted frame. We use these scenario spaces in several debugging exercises:

  • Slot Ambiguity: In Figure 3 (a), we investigate the scenario [IN:CREATE_ALARM [SL:ALARM_NAME ] [SL:DATE_TIME ]. Here, we notice a cluster of predictions with the frame [IN:CREATE_ALARM [SL:DATE_TIME ] missing the [SL:ALARM_NAME ]. These map to utterances such as “I want to wake up at 7 am” where the annotation has “wake up” is SL:ALARM_NAME; however, our model does not identify this. There are other examples, such as “wake me up at 7 am”, which are annotated without SL:ALARM_NAME, leading to ambiguity of whether or not “wake up” is an alarm name.

  • Incorrect Annotations: In Figure 3 (b), we investigate the scenario [IN:PLAY_MUSIC [SL:MUSIC_TYPE ]. Here, we notice a cluster of predictions with the frame [IN:PLAY_MUSIC [SL:MUSIC_GENRE ] [SL:MUSIC_TYPE ] adding the [SL:MUSIC_GENRE ]. These map to utterances such as “Play 1960s music” where here the annotation only has “music” as SL:MUSIC_TYPE, but our model predicts “1960s” as SL:MUSIC_GENRE. We believe this is an incorrect annotation in this cluster.

  • Underfitting: In Figure 3 (c), we investigate the scenario [IN:SET_DEFAULT_PROVIDER_MUSIC [SL:MUSIC_PROVIDER ]. This cluster is highly diverse, consisting of predictions from other music intents (e.g., IN:PLAY_MUSIC) and out-of-domain intents (e.g., IN:UNSUPPORTED_MUSIC). This scenario may need more data in order to be more properly defined.

7 Related Work

7.1 Efficient Decoding for Semantic Parsing

Recent trends in semantic parsing have started to shift towards the seq2seq paradigm Aghajanyan et al. (2020); Rongali et al. (2020); Li et al. (2021). However, seq2seq modeling typically has high latency, impeding application in real-world settings. One line of work aims at efficient parsing through logarithmic time decoding; for example, insertion parsing Zhu et al. (2020), bottom-up parsing Rubin and Berant (2021), and breadth-first parsing Liu et al. (2021). Alternative strategies include leveraging insertion transformers to recursively insert ontology tokens into the input, making the model’s decoding complexity in the number of intent-slot tokens Mansimov and Zhang (2021). Babu et al. (2021) introduces a one-shot CMLM Ghazvininejad et al. (2019) -style approach for non-autoregressive semantic parsing, where we model target tokens independently. However, such parses struggle in generalization due to the rigidity of the length prediction tasks. Shrivastava et al. (2021) further augment this with span-based decoding, leading to more consistent length prediction and, as a result, generalizable modeling.

Our work continues in the direction of one-shot decoding by leveraging span-based, non-autoregressive decoding, however, rather than relying on length prediction, we rely on scenario retrieval. We find scenario retrieval as a more interpretable and scalable intermediate task.

7.2 Scaling Semantic Parsing

Another critical property of semantic parses lies in reducing data requirements to stand up new domains and scenarios. Existing works rely on leveraging large language models such as BART

Lewis et al. (2020) with augmentations for scaling. In particular Chen et al. (2020b) introduce a meta-learning approach to improve domain scaling in the low-resource setting. Other works such as Liu et al. (2021); Zhu et al. (2020); Mansimov and Zhang (2021) aim to improve scaling through new decoding formulations. Desai et al. (2021) introduce the concept of intrinsic modeling where we provide a human-readable version of the semantic parsing ontology as context to encoding to improve few-shot generalization.

Our work leverages the intrinsic modeling paradigm by building a function to convert each intent-slot scenario into a readable representation via intent slot descriptions and example priming. Furthermore, our bi-encoder based retrieval setup allows us to inject additional context into each scenario and cache it to an index in order to retain inference efficiency.

7.3 Retrieval Based Semantic Parsing

Finally, there has been a recent trend towards dense retrieval in various NLP domains such as machine translation Cai et al. (2021), question answering Karpukhin et al. (2020), text generation Cai et al. (2019) and language modeling Borgeaud et al. (2018). Recent works also introduce retrieval-based semantic parsing Gupta et al. (2021); Pasupat et al. (2021). RetroNLU Gupta et al. (2021) and CASPER Pasupat et al. (2021) both leverage a retrieval step to provide examples as context to seq2seq models.

Our approach differs in two ways: (1) We phrase our problem as utterance-to-scenario retrieval rather than utterance-to-utterance retrieval. This allows us to look into supporting new scenarios with minimal-to-no-data required for retrieval. (2) Prior work leverage a separate module Pasupat et al. (2021) or separate iteration Gupta et al. (2021) for retrieval. We conduct our retrieval after encoding but prior to decoding as an intermediate step for non-autoregressive parsing. This allows our model to retain similar inference speed to one shot non-autoregressive decoding despite leveraging retrieval.

8 Conclusion

In this paper, we tackle scenario-based semantic parsing with retrieve-and-fill (RAF), a coarse-to-fine model which (a) retrieves a scenario with the best alignment to an utterance and (b) fills the scenario with utterance spans in leaf positions. Experiments show our model achieves strong results in high-resource, low-resource, and multilingual settings. The modular nature of our architecture also lends itself well to interpretability and debuggability; we perform several case studies uncovering the inner-workings of our approach.


  • A. Aghajanyan, J. Maillard, A. Shrivastava, K. Diedrick, M. Haeger, H. Li, Y. Mehdad, V. Stoyanov, A. Kumar, M. Lewis, and S. Gupta (2020) Conversational Semantic Parsing. In

    Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

    Cited by: §1, §2, §4.1, §4.2, §7.1.
  • A. Aghajanyan, A. Shrivastava, A. Gupta, N. Goyal, L. Zettlemoyer, and S. Gupta (2021) Better Fine-tuning by Reducing Representational Collapse. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: Appendix C, §3.3.
  • J. Andreas (2020) Good-Enough Compositional Data Augmentation. In Proceedings of the Annual Meeting of the Association for Compositional Linguistics (ACL), Cited by: §2.
  • A. Babu, A. Shrivastava, A. Aghajanyan, A. Aly, A. Fan, and M. Ghazvininej (2021) Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: §1, §3.2, §4.1, §4.2, §4.2, §7.1.
  • S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre (2018) Improving Language Models by Retrieving from Trillions of Tokens. arXiv preprint arXiv:2112.04426. Cited by: §7.3.
  • T. B. Brown, N. R. Benjamin Mann, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language Models are Few-Shot Learners. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), Cited by: §3.1.4.
  • D. Cai, Y. Wang, W. Bi, Z. Tu, X. Liu, and S. Shi (2019) Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Cited by: §7.3.
  • D. Cai, Y. Wang, H. Li, W. Lam, and L. Liu (2021) Neural Machine Translation with Monolingual Translation Memory. arXiv preprint arXiv:2105.11269. Cited by: §7.3.
  • T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020a) A Simple Framework for Contrastive Learning of Visual Representations. arXiv preprint arXiv:2002.05709. Cited by: §3.1.2.
  • X. Chen, A. Ghoshal, Y. Mehdad, L. Zettlemoyer, and S. Gupta (2020b) Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: Table 12, Table 13, Appendix B, Appendix C, §1, §1, §3.1.4, §4.1, Table 2, Table 4, §5.2, Table 6, Table 7, Table 8, §7.2.
  • N. Chomsky (1959) On certain formal properties of grammars. Elsevier. Cited by: §2.
  • A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020) Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: Appendix C, §4.2.
  • A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone, T. Lavril, et al. (2018) Snips Voice Platform: An Embedded Spoken Langauge Understanding System for Private-by-Design Voice Interfaces. arXiv preprint arXiv:1805.10190. Cited by: §1.
  • S. Desai and A. Aly (2021) Diagnosing Transformers in Task-Oriented Semantic Parsing. In Proceedings of the Findings of the Association for Computational Linguistics: ACL 2021, Cited by: §4.1.
  • S. Desai, A. Shrivastava, A. Zotov, and A. Aly (2021) Low-resource task-oriented semantic parsing via intrinsic modeling. arXiv preprint arXiv:2104.07224. Cited by: Figure 1, §1, §1, §3.1.3, §3.1.4, §4.1, §4.2, Table 3, §7.2.
  • L. Dong and M. Lapata (2018) Coarse-to-Fine Decoding for Neural Semantic Parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §3.
  • T. Gao, A. Fisch, and D. Chen (2020) Making Pre-trained Language Models Better Few-shot Learners. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL), Cited by: §3.1.4.
  • M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer (2019) Mask-Predict: Parallel Decoding of Conditional Masked Language Models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Cited by: §7.1.
  • S. Gupta, R. Shah, M. Mohit, A. Kumar, and M. Lewis (2018) Semantic Parsing for Task Oriented Dialog using Hierarchical Representations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1, §1, §2, §4.1, Table 2.
  • V. Gupta, A. Shrivastava, A. Sagar, A. Aghajanyan, and D. Savenkov (2021) RETRONLU: retrieval augmented task-oriented semantic parsing. arXiv preprint arXiv:2109.10410. Cited by: §4.1, §7.3, §7.3.
  • C. T. Hemphill, J. J. Godfrey, and G. R. Doddington (1990) The ATIS Spoken Language Systems Pilot Corpus. In Proceedings of the Workshop on Speech and Natural Language, Cited by: §1.
  • V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense Passage Retrieval for Open-Domain Question Answering. arXiv preprint arXiv:2004.04906. Cited by: §3.1.1, §3.1.2, §3.1.2, §7.3.
  • D. P. Kingma and J. Ba (2015) Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: Appendix C.
  • V. I. Levenshtein (1966) Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady 10, pp. 707. Cited by: Appendix A.
  • M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020) BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §1, §4.2, §7.2.
  • H. Li, A. Arora, S. Chen, A. Gupta, S. Gupta, and Y. Mehdad (2021) MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark. Proceedings of the European Chapter of the Association for Computational Linguistics (EACL). Cited by: Table 14, Table 15, Appendix B, §1, §4.1, Table 9, §7.1.
  • Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. Cited by: Appendix C, §4.2, §5.1.
  • Z. Liu, G. I. Winata, P. Xu, and P. Fung (2021) X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing. arXiv preprint arXiv:2106.03777. Cited by: §7.1, §7.2.
  • E. Mansimov and Y. Zhang (2021) Semantic Parsing in Task-Oriented Dialog with Recursive Insertion-based Encoder. arXiv preprint arXiv:2109.04500. Cited by: §1, §4.1, §4.2, §4.2, §7.1, §7.2.
  • B. Oğuz, K. Lakhotia, A. Gupta, P. Lewis, V. Karpukhin, A. Piktus, X. Chen, S. Riedel, W. Yih, S. Gupta, and Y. Mehdad (2021) Domain-matched pre-training tasks for dense retrieval. arXiv preprint arXiv:2107.13602. Cited by: §3.1.2.
  • I. Oren, J. Herzig, N. Gupta, M. Gardner, and J. Berant (2020) Improving Compositional Generalization in Semantic Parsing. In Proceedings of the Findings of the Association for Computational Linguistics (EMNLP), Cited by: §2.
  • P. Pasupat, Y. Zhang, and K. Guu (2021) Controllable semantic parsing via retrieval augmentation. arXiv preprint arXiv:2110.08458. Cited by: §7.3, §7.3.
  • G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton (2017)

    Regularizing Neural Networks by Penalizing Confident Output Distributions

    In Proceedings of the International Conference on Learning Representations (ICLR): Workshop Track, Cited by: §3.2.
  • S. Rongali, L. Soldaini, E. Monti, and W. Hamza (2020) Don’t Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing. In Proceedings of the Web Conference (WWW), Cited by: §1, §4.1, §7.1.
  • O. Rubin and J. Berant (2021) SmBoP: Semi-autoregressive Bottom-up Semantic Parsing. arXiv preprint arXiv:2010.12412. Cited by: §7.1.
  • T. Schick and H. Schütze (2021) It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. In Proceedings of the Conference on Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: §3.1.4.
  • T. Shin, Y. Razeghi, R. L. L. IV, E. Wallace, and S. Singh (2020) AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §3.1.4.
  • A. Shrivastava, P. Chuang, A. Babu, S. Desai, A. Arora, A. Zotov, and A. Aly (2021) Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing. In Proceedings of the Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: Figure 1, §1, §1, §3.2, §3.3, §4.1, §4.2, §4.2, §7.1.
  • J. Zhan, J. Mao, Y. Liu, J. Guo, M. Zhang, and S. Ma (2021) Optimizing Dense Retrieval Model Training with Hard Negatives. arXiv preprint arXiv:2104.08051. Cited by: §3.1.2.
  • Q. Zhu, H. Khan, S. Soltan, S. Rawls, and W. Hamza (2020) Don’t Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding. arXiv preprint arXiv:2010.03714. Cited by: §1, §7.1, §7.2.

Appendix A Heuristic Negatives

In order to understand the importance of model based hard negatives, we develop a simple heuristic to curate a set of hard negative scenarios for each gold scenario. Our heuristic involves selecting the top-N scenarios that share the top level intent but have the lowest Levenshtein edit distance Levenshtein (1966) compared to the gold scenario. The full algorithm is described in Algorithm 1.

In §5.1 and Table 6 we present the full results depicting the importance of model based negative sampling. We show that while our heuristic improves on top of no hard negatives (in-batch negatives only), it still lags behind model based hard negatives (-1.37%).

1:procedure Edit Distance Negatives
6:     for  do
8:               return heap
Algorithm 1 Heuristic-based negative sampling via edit distance.

Appendix B Intrinsic Descriptions

In table 11 we provide brief examples of the various representation functions used in RAF training.

Type Example
type-only [ intent [ slot ] ]
automatic-span [ add time timer [ measurement unit ] ]
automatic-type-span [ intent | add time timer [ slot | measurement unit ] ]
automatic-type-span-exs [ intent | add time timer [ slot | measurement unit | sec / min / hr ] ]
curated-span [ add time to timer [ unit of measurement ] ]
curated-type-span [ intent | add time to timer [ slot | unit of measurement ] ]
curated-type-span-exs [ intent | add time to timer [ slot | unit of measurement | sec / min / hr ] ]
Table 11: List of intrinsic representations used for encoding scenarios in RAF.

In tables 12, 13, 14, and 15 we present the intrinsic hand made descriptions used for each intent/slot on TOPv2 Chen et al. (2020b) and MTOP Li et al. (2021) respectively. §3.1.3 describes the various scenario representations used in full detail.

Intent Token Description
IN:ADD_TIME_TIMER add time to timer
IN:ADD_TO_PLAYLIST_MUSIC add music to playlist
IN:CANCEL_MESSAGE cancel message
IN:CREATE_ALARM create alarm
IN:CREATE_PLAYLIST_MUSIC create playlist music
IN:CREATE_REMINDER create reminder
IN:CREATE_TIMER create timer
IN:DELETE_ALARM delete alarm
IN:DELETE_REMINDER delete reminder
IN:DELETE_TIMER delete timer
IN:DISLIKE_MUSIC dislike music
IN:GET_ALARM get alarm
IN:GET_BIRTHDAY get birthday
IN:GET_CONTACT get contact
IN:GET_DIRECTIONS get directions
IN:GET_DISTANCE get distance between locations
IN:GET_ESTIMATED_ARRIVAL get estimated arrival time
IN:GET_ESTIMATED_DEPARTURE get estimated departure time
IN:GET_ESTIMATED_DURATION get estimated duration of travel
IN:GET_EVENT get event
IN:GET_EVENT_ATTENDEE get event attendee
IN:GET_EVENT_ATTENDEE_AMOUNT get amount of event attendees
IN:GET_EVENT_ORGANIZER get organizer of event
IN:GET_INFO_CONTACT get info of contact
IN:GET_INFO_ROAD_CONDITION get info of road condition
IN:GET_INFO_ROUTE get info of route
IN:GET_INFO_TRAFFIC get info of traffic
IN:GET_LOCATION get location
IN:GET_LOCATION_HOME get location of my home
IN:GET_LOCATION_HOMETOWN get location of my hometown
IN:GET_LOCATION_SCHOOL get location of school
IN:GET_LOCATION_WORK get location of work
IN:GET_MESSAGE get message
IN:GET_RECURRING_DATE_TIME get recurring date or time
IN:GET_REMINDER get reminder
IN:GET_REMINDER_AMOUNT get amount of reminders
IN:GET_REMINDER_DATE_TIME get date or time of reminder
IN:GET_REMINDER_LOCATION get location of reminder
IN:GET_SUNRISE get info of sunrise
IN:GET_SUNSET get info of sunset
IN:GET_TIME get time
IN:GET_TIMER get timer
IN:GET_TODO get todo item
IN:GET_WEATHER get weather
IN:HELP_REMINDER get help reminder
IN:IGNORE_MESSAGE ignore message
IN:LIKE_MUSIC like music
IN:LOOP_MUSIC loop music
IN:PAUSE_MUSIC pause music
IN:PAUSE_TIMER pause timer
IN:PLAY_MUSIC play music
IN:PREVIOUS_TRACK_MUSIC play previous music track
IN:REACT_MESSAGE react to message
IN:REMOVE_FROM_PLAYLIST_MUSIC remove from music playlist
IN:REPLAY_MUSIC replay music
IN:PREVIOUS_TRACK_MUSIC play previous music track
IN:REACT_MESSAGE react to message
IN:REMOVE_FROM_PLAYLIST_MUSIC remove from music playlist
IN:REPLAY_MUSIC replay music
IN:REPLY_MESSAGE reply to message
IN:RESTART_TIMER restart timer
IN:RESUME_TIMER resume timer
IN:SELECT_ITEM select item
IN:SEND_MESSAGE send message
IN:SEND_TEXT_MESSAGE send text message
IN:SET_DEFAULT_PROVIDER_MUSIC set default music provider
IN:SILENCE_ALARM silence alarm
IN:SKIP_TRACK_MUSIC skip music track
IN:SNOOZE_ALARM snooze alarm
IN:START_SHUFFLE_MUSIC start shuffling music
IN:STOP_MUSIC stop music
IN:SUBTRACT_TIME_TIMER subtract time from timer
IN:UNSUPPORTED_ALARM unsupported alarm request
IN:UNSUPPORTED_EVENT unsupported event request
IN:UNSUPPORTED_MESSAGING unsupported messaging request
IN:UNSUPPORTED_MUSIC unsupported music request
IN:UNSUPPORTED_NAVIGATION unsupported navigation request
IN:UNSUPPORTED_TIMER unsupported timer request
IN:UNSUPPORTED_WEATHER unsupported weather request
IN:UPDATE_ALARM update alarm
IN:UPDATE_DIRECTIONS update directions
IN:UPDATE_REMINDER update reminder
IN:UPDATE_REMINDER_DATE_TIMER update date time of reminder
IN:UPDATE_REMINDER_TODO update todo of reminder
IN:UPDATE_TIMER update timer
Table 12: List of intrinsic handmade descriptions for intents in TOPv2 Chen et al. (2020b).
Slot Token Description
SL:AGE age of person
SL:ALARM_NAME alarm name
SL:AMOUNT amount
SL:ATTENDEE attendee
SL:ATTENDEE_ADDED attendee to be added
SL:ATTENDEE_EVENT attendee of event
SL:ATTENDEE_REMOVED attendee to be removed
SL:ATTRIBUTE_EVENT attribute of event
SL:BIRTHDAY birthday
SL:CATEGORY_EVENT category of event
SL:CATEGORY_LOCATION category of location
SL:CONTACT contact
SL:CONTACT_RELATED contact related
SL:CONTENT_EMOJI content text with emoji
SL:CONTEnT_EXACT content text
SL:DATE_TIME date or time
SL:DATE_TIME_ARRIVAL date or time of arrival
SL:DATE_TIME_BIRTHDAY date or time of birthday
SL:DATE_TIME_DEPARTURE date or time of departure
SL:DATE_TIME_NEW new date or time
SL:DATE_TIME_RECURRING recurring date or time
SL:DESTINATION travel destination
SL:DURATION duration
SL:FREQUENCY frequency
SL:GROUP group
SL:JOB job
SL:LOCATION location
SL:LOCATION_CURRENT current location
SL:LOCATION_HOME location of my home
SL:LOCATION_MODIFIER location modifier
SL:LOCATION_USER location of user
SL:LOCATION_WORK location of work
SL:MEASUREMENT_UNIT unit of measurement
SL:METHOD_RETRIEVAL_REMINDER method of retrieving reminder
SL:METHOD_TIMER method of timer
SL:METHOD_TRAVEL method of traveling
SL:MUSIC_ALBUM_TITLE title of music album
SL:MUSIC_ARIST_NAME name of music artist
SL:MUSIC_GENRE genre of music
SL:MUSIC_PLAYLIST_TITLE title of music playlist
SL:MUSIC_PROVIDER_NAME name of music provider
SL:MUSIC_RADIO_ID id of music radio
SL:MUSIC_TRACK_TITLE title of music track
SL:MUSIC_TYPE type of music
SL:MUTUAL_EMPLOYER mutual employer
SL:MUTUAL_LOCATION mutual location
SL:MUTUAL_SCHOOL mutual school
SL:NAME_APP name of app
SL:NAME_EVENT name of event
SL:OBSTRUCTION_AVOID obstruction to avoid
SL:ORDINAL ordinal
SL:ORGANIZER_EVENT obstruction to avoid
SL:ORDINAL ordinal
SL:ORGANIZER_EVENT organizer of event
SL:PATH path
SL:PATH_AVOID path to avoid
SL:PERIOD time period
SL:PERSON_REMINDED person to be reminded
SL:PERSON_REMINDED_ADDED added person to be reminded
SL:PERSON_REMINDED_REMOVED removed person to be reminded
SL:POINT_ON_MAP point on map
SL:RECIPIENT message recipient
SL:RECURRING_DATE_TIME recurring date or time
SL:RECURRING_DATE_TIME_NEW new recurring date or time
SL:RESOURCE resource
SL:ROAD_CONDITION road condition
SL:ROAD_CONDITION_AVOID road condition to avoid
SL:SEARCH_RADIUS search radius
SL:SENDER message sender
SL:SOURCE travel source
SL:TAG_MESSGE tag of message
SL:TIMER_NAME timer name
SL:TIME_ZONE time zone
SL:TODO todo item
SL:TODO_NEW new todo item
SL:TYPE_CONTACT contact type
SL:TYPE_CONTENT content type
SL:TYPE_INFO info type
SL:TYPE_REACTION reaction type
SL:TYPE_RELATION relation type
SL:UNIT_DISTANCE unit of distance
SL:WAYPOINT waypoint
SL:WAYPOINT_ADDED waypoint to be added
SL:WAYPOINT_AVOID waypoint to avoid
SL:WEATHER_ATTRIBUTE weather attribute
Table 13: List of intrinsic handmade descriptions for slots in TOPv2 Chen et al. (2020b).
Intent Token Description
IN:FOLLOW_MUSIC follow music
IN:GET_JOB get job
IN:GET_GENDER get gender
IN:GET_UNDERGRAD get undergrad education
IN:GET_MAJOR get college major
IN:GET_EDUCATION_DEGREE get education degree of person
IN:GET_AGE get age of person
IN:DISPREFER dislike item
IN:RESUME_MUSIC resume music
IN:QUESTION_MUSIC question about music
IN:CREATE_CALL create a caoo
IN:GET_AIRQUALITY get airquality
IN:GET_CALL_CONTACT get contact for caller
IN:SET_UNAVAILABLE set status to unavailable
IN:END_CALL end call
IN:STOP_SHUFFLE_MUSIC stop shuffle of music
IN:PREFER prefer item
IN:GET_LANGUAGE get language
IN:SET_AVAILABLE set available
IN:GET_GROUP get group
IN:ANSWER_CALL answer call
IN:GET_CONTACT_METHOD get method to contact
IN:UPDATE_METHOD_CALL update method of call
IN:GET_ATTENDEE_EVENT get attendee for event
IN:UPDATE_CALL update call
IN:GET_LIFE_EVENT get life event
IN:REPEAT_ALL_MUSIC repeat all music
IN:GET_EDUCATION_TIME get education time
IN:QUESTION_NEWS question about news
IN:GET_EMPLOYER get employer
IN:IGNORE_CALL ignore call
IN:REPEAT_ALL_OFF_MUSIC turn of repeat
IN:UNLOOP_MUSIC turn loop off
IN:SET_DEFAULT_PROVIDER_CALLING set default provider for calling
IN:GET_AVAILABILITY get avalability of contact
IN:HOLD_CALL hold call
IN:GET_LIFE_EVENT_TIME get time of life event
IN:SHARE_EVENT share event
IN:CANCEL_CALL cancel call
IN:SET_RSVP_YES set rsvp to yes
IN:PLAY_MEDIA play media
IN:GET_TRACK_INFO_MUSIC get information about the current track
IN:GET_DATE_TIME_EVENT get the date time of the event
IN:SET_RSVP_NO set rsvp to no
IN:MERGE_CALL marge call
IN:UPDATE_REMINDER_LOCATION update the location of the reminder
IN:GET_MUTUAL_FRIENDS get mutual friends
IN:GET_MESSAGE_CONTACT get information about message contact
IN:GET_LYRICS_MUSIC get lyrics about the song
IN:GET_INFO_RECIPES get information about recipe
IN:GET_DETAILS_NEWS get news details
IN:GET_EMPLOYMENT_TIME get employment time
IN:GET_RECIPES get a recipe
IN:GET_CALL get call
IN:GET_CALL_TIME get time of the call
IN:GET_CATEGORY_EVENT get the category of the event
IN:RESUME_CALL resume the call
IN:IS_TRUE_RECIPES ask question about recipes
IN:SET_RSVP_INTERESTED set rsvp to interested
IN:GET_STORIES_NEWS get news stories
IN:SWITCH_CALL switch call
IN:REWIND_MUSIC rewind the song
IN:FAST_FORWARD_MUSIC forward the song
Table 14: List of intrinsic handmade descriptions for intents in MTOP Li et al. (2021).
Slot Token Description
SL:GENDER gender of person
SL:RECIPES_TIME_PREPARATION time to prepare recipe
SL:RECIPES_EXCLUDED_INGREDIENT exclude ingredient for recipe
SL:USER_ATTENDEE_EVENT attendee of event
SL:MAJOR major
SL:RECIPES_TYPE type of recipe
SL:SCHOOL school
SL:TITLE_EVENT title of event
SL:RECIPES_DISH recipe dish
SL:NEWS_TYPE type of news
SL:RECIPES_SOURCE source of recipe
SL:RECIPES_DIET diet of recipe
SL:RECIPES_UNIT_NUTRITION nutrition unit of recipe
SL:MUSIC_REWIND_TIME time to rewind music
SL:RECIPES_TYPE_NUTRITION nutrition type of recipe
SL:CONTACT_METHOD method to contact
SL:SIMILARITY similarity
SL:PHONE_NUMBER phone number
SL:NEWS_CATEGORY category of news
SL:EDUCATION_DEGREE education degree
SL:RECIPES_RATING rating of recipe
SL:CONTACT_REMOVED removed contact
SL:NEWS_REFERENCE news reference
SL:METHOD_RECIPES method of recipe
SL:LIFE_EVENT life event
SL:RECIPES_MEAL recipe meal
SL:NEWS_TOPIC news topic
SL:RECIPES_ATTRIBUTE recipe attribute
SL:EMPLOYER employer
SL:RECIPES_COOKING_METHOD cooking method of recipe
SL:RECIPES_CUISINE cuisine of recipe
SL:MUSIC_PLAYLIST_MODIFIER music playlist modifier
SL:RECIPES_QUALIFIER_NUTRITION nutrition qualifier of recipe
SL:METHOD_MESSAGE method to send message
SL:RECIPES_UNIT_MEASUREMENT unit of measurement in recipe
SL:CONTACT_ADDED added contact
SL:NEWS_SOURCE news source
Table 15: List of intrinsic handmade descriptions for slots in MTOP Li et al. (2021).

Appendix C Hyperparameters

In this section we describe the hyper parameters for training our various RAF models.

Architecture Parameters.

For our RAF architectures we leverage a shared RoBERTa Liu et al. (2019) or XLM-R Conneau et al. (2020) encoder for both the utterance and scenario encoders. We augment each of these encoders with an additional projection layer with a hidden dimension of 768 (base models) or 1024 (large models). For our span pointer decoder we leverage a 1 layer transformer decoder with the same hidden dimension as the respective encoder (1L, 768/1024H, 16/24A).

Optimization Parameters.

We train our models with the Adam Kingma and Ba (2015) optimizer along with a warmup and linear decay. We train our models across 8 GPUs with 32GB memory each. Additionally we optionally augment our models with the R3F loss Aghajanyan et al. (2021)

based on validation set tuning in each setting. To determine hyperparameters, we conduct hyperparameter sweeps with 56 iterations each. The hyperparameters for the high resource runs on TOPv2

Chen et al. (2020b) are described in Table 16.

Parameter RAF Models
Epochs 40
Optimizer Adam
Weight Decay 0.01
Warmup Period (steps) 1000
Learning Rate Scheduler Linear Decay
Learning Rate 0.00002 0.00003 0.00002
Batch Size 40 12 16
2.69 4 2.69
# GPU 8
GPU Memory 32GB 32GB 32GB
Table 16: Hyperparameter values for RAF architectures.