On Codex Prompt Engineering for OCL Generation: An Empirical Study

03/28/2023
by   Seif Abukhalaf, et al.
0

The Object Constraint Language (OCL) is a declarative language that adds constraints and object query expressions to MOF models. Despite its potential to provide precision and conciseness to UML models, the unfamiliar syntax of OCL has hindered its adoption. Recent advancements in LLMs, such as GPT-3, have shown their capability in many NLP tasks, including semantic parsing and text generation. Codex, a GPT-3 descendant, has been fine-tuned on publicly available code from GitHub and can generate code in many programming languages. We investigate the reliability of OCL constraints generated by Codex from natural language specifications. To achieve this, we compiled a dataset of 15 UML models and 168 specifications and crafted a prompt template with slots to populate with UML information and the target task, using both zero- and few-shot learning methods. By measuring the syntactic validity and execution accuracy metrics of the generated OCL constraints, we found that enriching the prompts with UML information and enabling few-shot learning increases the reliability of the generated OCL constraints. Furthermore, the results reveal a close similarity based on sentence embedding between the generated OCL constraints and the human-written ones in the ground truth, implying a level of clarity and understandability in the generated OCL constraints by Codex.

READ FULL TEXT
research
05/31/2023

Measuring the Robustness of Natural Language Processing Models to Domain Shifts

Large Language Models have shown promising performance on various tasks,...
research
05/23/2022

What Makes Data-to-Text Generation Hard for Pretrained Language Models?

Expressing natural language descriptions of structured facts or relation...
research
03/09/2022

HealthPrompt: A Zero-shot Learning Paradigm for Clinical Natural Language Processing

Deep learning algorithms are dependent on the availability of large-scal...
research
03/14/2023

Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences

As a natural language assistant, ChatGPT is capable of performing variou...
research
02/16/2022

ZeroGen: Efficient Zero-shot Learning via Dataset Generation

There is a growing interest in dataset generation recently due to the su...
research
05/23/2023

Prompt position really matters in few-shot and zero-shot NLU tasks

Prompt-based models have made remarkable advancements in the fields of z...
research
05/24/2023

Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing

In executable task-oriented semantic parsing, the system aims to transla...

Please sign up or login with your details

Forgot password? Click here to reset