Phishing is a type of cyber-attack whereby criminals design seemingly authentic emails with the intent of tricking users into giving up private, confidential, and/or sensitive information such as login passwords and financial data. Artificial Intelligence-based automated phishing detection methods are limited as Machine Learning (ML) models can only identify the malicious patterns they have been trained on and detection by heuristics-based techniques produce high false positive rates(Basit et al., 2021). In today’s highly connected world, users spend considerable time sending and reading both work and personal emails, and it is thus hard for users to pay close attention to every email. Consequently, they can fall victim to advanced deceptive phishing techniques that bypass security filters.
Recent work shifts the focus from automated phishing detection to detection support to assist users in making their own judgements (Althobaiti et al., 2021), since automated approaches are not always 100% accurate and real-time support such as warnings can effectively change risky behaviors. Presenting users with security indicators’ information enables human strength in capturing abnormal behaviors, such as contextual awareness.A human-centric solution using autonomous ML agents to aid judgment can therefore be a crucial step in the right direction. Despite security warnings and modern means of educating users about phishing, current detection support and training still cannot deal with sophisticated malicious emails in real time. Therefore, condensing important email information relevant to phishing and rationalising human efforts are important objectives to deal with new types of attacks.
Most of the existing works on phishing detection support heavily rely on the legitimacy of URLs (Alsharnouby et al., 2015; Althobaiti et al., 2021). However, explanations of URL features are easy to understand by general users, e.g., website rank, hostname, and domain popularity. More importantly, phishing attacks have evolved to escape URL detection by leveraging social engineering. In particular, criminals can generate links through third-party services or a plaintext with a response request (Almashor et al., 2021). Motivated by that, we aim to address the challenges of improving the readability and effectiveness of generated information on phishing emails that URL-based detection methods cannot detect.
Therefore, the objective of this work is to design and develop a human-centric notification system based on both emails and their contexts using Natural Language Processing (NLP) methods to help users accurately identify phishing attempts. The intention is to outline useful information based only on the email and the contextual knowledge that machines can utilize - to be precise, this includes the summary of an email, the emotions associated with an email and the intents of the sender that may be relevant to evoking self-sabotaging actions.
We present a novel, human-centric mechanism to combat one of the most prevalent cybercrimes of the world today - email phishing. Our contributions to that end can be summarized as follows:
We propose a system to pipeline emails through ML models and to generate a human-centric summary report on useful criteria for the user to identify phishing.
We leverage and investigate the utility of the summarization capability of the latest transformer-based ML models on emails for phishing analysis.
We leverage and investigate the utility of the emotion classification capability of transformer-based ML models on emails in the context of “cognitive triaging” to detect potential psychological triggers.
We leverage and investigate the utility of the intent analysis capability of transformer-based ML models on emails to determine whether malicious intent can be accurately “cherry-picked”.
2. System Design
We identified three critical components in particular as utilities of state-of-the-art ML models that could indicate potential phishing emails, and, as a result, we build our system to combine them. The system design thus incorporates three fundamental pipelines: extractive summarization, emotion classification, and intent analysis. We next explain those three pipelines.
Extractive Summarization. Phishing emails can be overwhelmingly long to deceive the recipients to overlook the phishing components and to focus on urgency and/or emotions. To emphasize the most important context, we use the summarization capability of modern ML models. Informing the user of the most important contents of the email and of the potentially desired actionable items evoked by the sender could allow them to examine the primary purpose of the email without any superfluous or distracting information clouding their judgement. For instance, if a small part of the email evokes a potential victim to click on a harmful link whereas the others are merely warnings of what would occur if they don’t, an ideal summarization branch/pipeline would immediately point this out as as irregular as part of the user results.
Emotion Classification. The multi-class emotion classification branch is inspired by the brilliant paper on cognitive triaging (Van Der Heijden and Allodi, 2019). There are six possible “cognitive triggers” we detect in phishing emails - Reciprocity, Consistency, Social Proof, Authority, Liking, and Scarcity (Van Der Heijden and Allodi, 2019). The definition of these terms are, respectively, as follows: Reciprocity - Tendency to feel obliged to repay favors from others. Consistency - Tendency to behave in a way consistent with past decisions and behaviors. Social Proof - Tendency to reference the behavior of others to guide one’s own actions. Authority - Tendency to obey people in authoritative positions. Liking - Preference for saying “yes” to the requests of people one knows and likes. Scarcity - Tendency to assign more value to items and opportunities when their availability is limited (Van Der Heijden and Allodi, 2019).
Intent Analysis. This analysis filters and detects potentially malicious intent in the form of actionable and identifiable items that can be detected through the use of ML models. The aim of this branch is that a recipient gets a clearer picture as to whether an email is harmful when highlighted and contextualized suspicious portions of an email are taken into consideration with information from the other branches. Highlighting potential phishing intent could also serve to train recipients to develop a “feel” for what phishing intents and signals look like.
3. Preliminary results
3.1. Data collection and pre-processing
The first experimental setup step was to compile email datasets which include the Cambridge phishing dataset, the Cornell phish bowl, the Enron email corpus of benign emails, the millersmiles.co.uk phishing dataset, and the Nazario phishing dataset. The above were chosen as they were collected by reputable researchers to create an accurate representation of phishing contexts or of benign correspondence in emails - in fact, all datasets in question compile real-life emails. The corpora contain 41446, 1757, 252721, 33080, and 946 emails respectively, and were all utilized to create random experiment samples of varying sizes in a variety of contexts - early evaluation sample sizes numbered 500 or more emails followed by final evaluation samples numbering 5000 or more for all branches.
For all three branches, pre-processing separated the email body from the rest of the email and removed HTML tags if they were present. In addition, pre-processing for the intent analysis and emotion classification pipelines also included the splitting of email bodies into separate sets of words each with an associated set of labels such as relevant emotions or intent.
Recently, the advent of modern leading ML models has been propagated and bolstered largely thanks to the release of the “Transformer” by researchers at Google (Vaswani et al., 2017). As a consequence of its release, current leading models in terms of the universality of their applicability in the ML space such as T5 (Raffel et al., 2020), XLNet (Yang et al., 2019), and BERT (Devlin et al., 2019) all integrate the Transformer to varying degrees. Such universality means that they can be applied in a variety of different contexts, including emails, to obtain impressive results.
Extractive Summarization. We build a summarization algorithm that leverages recent transformer-based models: T5 (Raffel et al., 2020), XLNet (Yang et al., 2019), and BERT (Devlin et al., 2019). T5 has produced the most promising results and, as a result, it has been adopted for this task. T5 has the advantage that it does not need to be fine-tuned for individual tasks by design. As a result, the implementation of the task and purpose with this model has been somewhat trivial. Whereas the utility of achieving the above may be evident, the precise manner in which content is to be summarized in an ideal scenario is of course open to interpretation since there are no objective criteria. As a result, fractional limits have been set with regards to the word length of a generated summary for any given email for examination in addition to hard-set limits such as 25, 50, or 100 words. Closer examination and some metric such as cross-validation is potentially needed to decide on how to best ascertain what word limit the summary should have - through admittedly looser criteria such as subjectively judging grammar, readability, and the ratio of summary length to essential context, the most promising result has been to set the maximal length of a summary to be a fraction of one fifth of the length of the original email.
. We first curated a random sample of sentences from 500 emails to be labelled under seven classes, with the additional class “None” represenitng that no given quality was present. This labelled set was then used to train the T5 and BERT models such that an individual set of words from any given email could have its most likely prevalent qualities identified by analyzing logit score outputs from the models implemented. The overall assessment of the sentences and sets of words in the emails was then used to build a combined probability score that the email has a “spike” in one or more of the six cognitive triaging qualities - indicating a likely attempt to phish. Combined with assessments from the other branches the intent is to supply a user with all the prospective information they may need to make a rational and informed assessment themselves given what emotions are present and what they are trying to elicit in the context of the email - information generated with the help of this branch.
Intent Analysis. The results have shown that the most promising models to implement this branch have been T5 and BERT, with T5 edging out BERT slightly with regards to accuracy and loss. To categorize common phishing intents that were present in emails, only short and apt descriptions have been added as tags to sets of word - such as “click link” or “download file”. We randomly selected 500 emails from our datasets and conducted tagging manually. We then trained T5 to recognize malicious intents in the sentences of any email, with the objective of “cherry-picking” phishing intent in new emails users should pay particular attention to if other indicators such as cognitive triaging or the summary are suspicious. Figure 2 shows examples of intent evocations. We can note that the generated, concise summary contains the triggers and intent evocations at a much higher density, as we have suspected given our description in Section 2. By informing the user of the threshold criteria by which the tags are generated and the prospective danger of there being too many, the ambition is to achieve our set out objective of keeping the user alert. Experimentation at a larger scale is the next planned step to optimize this branch further.
4. Concluding Remarks
In this work, we devised a human-centric notification mechanism that extracts prospective psychological triggers, possible malicious intent, and a representative summary from emails. We then present the above in a meaningful way to the user for better decision-making and to elevate their learning of continuously evolving phishing patterns. Further examination with user studies and objective experimental analysis at a larger scale will give more insight into the effectiveness of this methodology. A concertized metric of trigger and intent density as a fraction of the total email length can also be examined to determine the metrics’ correlation with whether an email is “phishy” or not.
- Characterizing malicious url campaigns. arXiv preprint arXiv:2108.12726. Cited by: §1.
- Why phishing still works: user strategies for combating phishing attacks. International Journal of Human-Computer Studies 82, pp. 69–82. Cited by: §1.
- I don’t need an expert! making url phishing features human comprehensible. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, pp. 1–17. External Links: Cited by: §1, §1.
- A comprehensive survey of ai-enabled phishing attacks detection techniques. Telecommunication Systems 76 (1), pp. 139–154. Cited by: §1.
- BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171–4186. External Links: Cited by: §3.2, §3.2.
Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), pp. 1–67. External Links: Cited by: §3.2, §3.2.
- Cognitive triaging of phishing attacks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 1309–1326. Cited by: §2.
- Attention is all you need. Advances in neural information processing systems 30. Cited by: §3.2.
- Xlnet: generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32. Cited by: §3.2, §3.2.