Predicting job-hopping likelihood using answers to open-ended interview questions

07/22/2020 ∙ by Madhura Jayaratne, et al. ∙ 0

Voluntary employee turnover incurs significant direct and indirect financial costs to organizations of all sizes. A large proportion of voluntary turnover includes people who frequently move from job to job, known as job-hopping. The ability to discover an applicant's likelihood towards job-hopping can help organizations make informed hiring decisions benefiting both parties. In this work, we show that the language one uses when responding to interview questions related to situational judgment and past behaviour is predictive of their likelihood to job hop. We used responses from over 45,000 job applicants who completed an online chat interview and also self-rated themselves on a job-hopping motive scale to analyse the correlation between the two. We evaluated five different methods of text representation, namely four open-vocabulary approaches (TF-IDF, LDA, Glove word embeddings and Doc2Vec document embeddings) and one closed-vocabulary approach (LIWC). The Glove embeddings provided the best results with a positive correlation of r=0.35 between sequences of words used and the job-hopping likelihood. With further analysis, we also found that there is a positive correlation of r=0.25 between job-hopping likelihood and the HEXACO personality trait Openness to experience. In other words, the more open a candidate is to new experiences, the more likely they are to job hop. The ability to objectively infer a candidate's likelihood towards job hopping presents significant opportunities, especially when assessing candidates with no prior work history. On the other hand, experienced candidates who come across as job hoppers, based purely on their resume, get an opportunity to indicate otherwise.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Voluntary turnover, which represents the vast majority of all employee turnover, decreases organizational productivity and dampen employee morale [1] while inflicting direct financial costs such as sourcing, recruiting and onboarding costs. However making frequent voluntary job changes, known as job-hopping, has become a trend in the recent past [2]. The motivations for job-hopping, have been identified to be two-fold; advancement and escape [3]. The advancement motive represents the growth and career perspective, while the escape motive represents a withdrawal or dislike of the work environment, especially among those who are described as impulsive and unpredictable. The latter is identified as a psychological property and commonly known as the “hobo syndrome” [4]. Further studies have shown the relationship between personality and voluntary turnover [5, 6, 7, 8, 9].

The ability to assess a candidate’s likelihood of job-hopping prior to selection can help both candidates and employers make better decisions and avoid future surprises and costs due to voluntary exits. The most frequently used approach to discover patterns of job-hopping is to explore the employment history listed in an applicant’s resume. Sifting through resumes can be both time-consuming and unreliable, especially in situations of high volume recruitment. Resumes are also known to produce biased outcomes [10, 11]. Moreover, it is an ineffective method with novice job seekers, such as new graduates who have insignificant job histories.

Given the demonstrated link between one’s personality and voluntary turnover, in this study, we evaluate whether the answers given by candidates to interview questions related to past behaviour and situational judgement can be used to infer their job-hopping likelihood. The basis for selecting interview answers as a possible predictor is two-fold. Firstly, one’s language use has been shown to be highly predictive of their personality. Personality traits have been successfully derived from informal (microblogs [12, 13, 14], social media posts [15, 16]), semiformal (blogs [17], interview questions [18]) and formal (essays [19]) contexts. Authors own prior work has shown that interview answers are a strong predictor of personality traits [18]. Secondly, structured interviews where the same questions are asked from every candidate in a controlled conversation flow and evaluated using a well-defined rubric have shown to reduce bias [20] and also increase the ability to predict future job performance [21]. Computational inference of job-hopping likelihood from interview responses (Figure 1) further increases the utility of the structured interview and its applicability in high volume recruitment.

Figure 1: Using interview responses to predict candidates’ job-hopping likelihood

In this work, we make the following contributions to the crossroads of computational linguistics and organizational psychology domains.

  1. We demonstrate that responses to typical interview questions related to past behaviour and situational judgement can be used to reliably infer one’s job-hopping likelihood.

  2. We evaluate multiple methods of text representations and establish that the Glove based word-embedding method achieves the highest correlation of r=0.35 between text and job-hopping likelihood when used with a Random Forest regressor.

  3. We validate the positive correlation between job-hopping motive and Openness to experience (one of the personality traits in the HEXACO personality model), both derived from text (r=0.25). This is in line with previous findings using standard personality tests.

The rest of the paper is organised as follows. Section 2 presents a detailed background into the research on employee turnover, the role of personality on turnover and the link between language and personality. In section 3, we describe the methodology, including the data used and the five different text representation methods we evaluated, namely TF-IDF, LDA, Glove word embeddings, Doc2Vec document embeddings and LIWC. Results, in terms of the accuracies achieved by each text representation method, are presented in section 4 along with discussion and further analysis of salient correlations, demographics, and terms used. Section 5 concludes the paper with a summary and future research directions.

2 Background

A study conducted by the Australian HR Institute in 2018 across all major industry sectors in Australia [22] found that on average companies face an annual turnover rate of 18% and within the age group of 18 to 35 it jumps to 37%. That is more than 1 in 3 people in the youngest age group leaving an organization within a year. A majority (63%) of respondents in the study, mostly HR staff, claimed that their organisation does not measure the financial cost of employee turnover. Employee turnover rate is much higher than the average for some industries such as hospitality [23, 1]. Cho et al. [23] report a staggering 115% turnover rate among non-managerial employees of hospitality firms while for managerial employees it is 35%.

Majority of employee turnover consists of voluntary turnover, that is, employee-initiated separations compared to involuntary turnover initiated by the employer such as layoffs and terminations due to poor performance. Significant costs have to be borne by an organization when an employee voluntarily leaves. These include replacement costs such as costs associated with advertising, screening and selecting a new candidate, employee training costs, and operational efficiency losses until a new employee reaches a sufficient level of productivity. These costs are exacerbated by the dampening of remaining employee morale leading to lower quality work and lower productivity [1]. A study conducted by the Work Institute in the US [24]

found that voluntary exit of an employee costs a company 33% of the employee’s base salary, which the authors claim is a conservative estimate. The report also states that with a median base salary of $45,000, it is costing the US economy close to $600bn a year due to voluntary turnover. Similarly, in a large-scale meta-analysis (N

300,000), Park and Shaw [25] observed a significant and negative () correlation between voluntary turnover rates on organizational performance.

Voluntary turnover has been associated with a number of negative job attributes such as low level of job satisfaction, lack of promotion opportunities, lack of work-life balance, lack of fairness of the firm’s procedures etc. [26], which are reasons originating from a misalignment between employee and employer expectations. Measures can be put in place by the employer to discover and address these issues where possible and methods such as employee engagement surveys and periodic review discussions serve this purpose.

The focus of our work is a type of voluntary turnover identified as “job hopping”, the frequent move from job to job [3] shown by some individuals than others. Studies have shown that there is a relationship between one’s personality and job-hopping motives [5, 6, 27, 7, 8, 9, 28, 29]. The hypothesis behind these studies is that one’s personality plays a key role in their intention to hop jobs and acts as a latent variable that mediates their desire to voluntarily leave an organization. Barton and Cattell [5]

conducted one of the earliest longitudinal studies on the effect of personality on job promotion and job change and found that individuals who were more

practical and down to earth recorded the lowest incidents on job turnover. Ghiselli [4] named this tendency the “hobo syndrome”, an internal impulse-driven move from one job to another shown by some employees irrespective of other more rational motives. Lake et al [3] refer to this as an “escape motive” or a sudden withdrawal from the work environment as opposed to an “advancement motive” that makes an employee leave for a perceived better opportunity. More recent studies have also shown a correlation between hobo syndrome and the Big 5 personality trait of Openness to experience [30] further validating the personality influence on job-hopping motives. Other studies have shown similar relationships between the Big-5 personality traits and turnover intention. For instance, Sarwar et at. [29] found personality traits Extraversion, Neuroticism, Conscientiousness and Agreeableness to be negatively correlated with turnover intention while Openness to experience was positively related with turnover intention. These results are in line with the Big-5 personality traits’ correlations with the intention to quit observed in a meta-analysis conducted by Zimmerman [9]. In a study conducted with call centre employees, Timmerman [8] reported slightly different results with Neuroticism, Conscientiousness and Agreeableness as negatively correlated with turnover while Extraversion and Openness to experience positively correlated with turnover. With the above studies, it is pertinent to conclude that overall there is a latent relationship between one’s personality and his/her turnover intentions.

Assessing one’s job-hopping likelihood is currently achieved via self-rating questionnaires similar to standard personality tests. One such validated self-rating item list is the Job-Hopping Motive Scale developed by Lake, Highhouse and Shrift [3]. It includes eight self-rating items with four items each validated with factor analysis to assess escape and advancement motives. Their study also confirms the positive correlation of escape motive with impulsivity (r=0.19) and a negative correlation with persistence (r=-0.16). However a challenge with administering self-rating based assessments is the need to have multiple statements to gain a measure of a single personality construct (for example four items are required to measure escape motive). It is not unusual to have over 100 items in such tests when you combine other measures such as personality traits. Applicant reactions to such personality tests have shown to be less favourable than interviews [31, 32]. Based on a meta-analysis of multiple studies on applicant reaction to selection methods, Anderson et al. [31] found that compared to job interviews and work sample tests, personality tests fall short of making a positive impression with candidates in areas of face validity, opportunity to perform, interpersonal warmth and respectful of privacy. These indicate candidates’ preference to express themselves and not be restricted to self-rating themselves on a pre-defined set of multiple-choice questions (typically over 100 items) as found in standard personality tests.

On the other hand, researchers have demonstrated that one’s language use is indicative of his/her personality attributes [33, 34, 35, 36, 37]. This allows structured interviews, which are much more engaging for applicants and permit open expression of applicant thoughts to be used as a source for inferring personality attributes. Authors have demonstrated in [18] how responses to open-ended interview questions can be used to reliably infer one’s personality attributes based on the six-factor HEXACO model [38].

Combining the relationship between job-hopping likelihood and personality with that of personality and language use, we hypothesised that one’s language use is closely associated with their job-hopping motives and hence the job-hopping likelihood can be inferred from various characteristics of their language use. Based on this hypothesis, we envisioned building a language-based model that is able to accurately predict a candidate’s job-hopping likelihood from answers to regular open-ended interview questions. Such a model would have wide applicability in digitised forms of interviews, be it chat-based, voice or video where the textual content of the candidate answers can be used to infer job-hopping likelihood in addition to the personality and communication skills that can be derived from text. This allows the use of digitised interviews, which are much more engaging than standard personality tests and preferred by candidates to be used as a multi-measure assessment scalable to high applicant volumes supported by algorithmic inferences.

3 Methodology

In order to test the correlation between language use and job-hopping likelihood, we built a regression model that is able to infer the job-hopping motive scale rating (discussed above) using textual answers given to open-ended interview questions. Given the importance of numerical representation of language in building a machine learning model, we compared the performance of five different text representation methods namely, terms (TF-IDF), topics (LDA), Glove word embeddings, Doc2Vec and LIWC. In this section, we describe the training dataset, the five different text representation methods and the regression model building approach.

3.1 Data

We analysed free-text responses from 45,899 candidates who used the PredictiveHire111https://www.predictivehire.com/ FirstInterviewTM platform, an online chat-based interview tool. Job applicants answer 5-7 open-ended questions and self-rating questions based on a proprietary personality inventory that also included the Job-Hopping Motives Scale items designed by Lake et al. [3]. FirstInterviewTM is typically the very first engagement the applicant has with the hiring organisation, placed at the top of the recruitment funnel and close to 40% of applicants complete it on a mobile.

The online interview questionnaire includes open-ended free-text questions on past experience, situational judgement and values. The questions are customisable by role family (e.g. sales, retail, call centre etc.) and specific customer value requirements. The questions are rotated regularly to address gaming risk. Following are some example questions.

  • What motivates you? What are you passionate about?

  • Not everyone agrees all the time. Have you had a peer, teammate or friend disagree with you? What did you do?

  • Give an example of a time you have gone over and above to achieve something. Why was it important for you to achieve this?

  • Sometimes things don’t always go to plan. Describe a time when you failed to meet a deadline or personal commitment. What did you do? How did that make you feel?

  • In sales, thinking fast is critical. What qualifies you for this? Provide an example.

Following are two example answers to two of the free text questions.

  • What motivates you? What are you passionate about?

    • There is a process to everything. Whether it’s making phone calls or cooking. I like to always do my best and learn the process.

    • Early retirement by paying off my home early. Literally most of my focus is to pay off my house in another 8 years. The better the results, the better the commissions.

  • Sometimes things don’t always go to plan. Describe a time when you failed to meet a deadline or personal commitment. What did you do? How did that make you feel?

    • My first month, I failed to meet my end of month target. I am a competitive person so I did feel a bit sad and defeated but after a great coaching session I was able to meet and exceed targets 7 months in a row.

    • I haven’t experienced a time where I didn’t meet a deadline. I always assure I do any task given to me.

The length of textual responses in terms of words had a and .

The Job-Hopping Motives Scale items consist of the following eight statements where each candidate self-rated themselves using a 5-point scale ranging from Strongly Agree to Strongly Disagree.

  • Because working for one company tends to create boredom, people should move from company to company often.

  • Even if someone has changed jobs several times, they should take a new job if it involves moving to a better position.

  • Frequently moving between jobs is perfectly justified when each job change leads to a more impressive job.

  • When a person discovers they dislike their coworkers, they should move to another job, and keep switching jobs until they finally find a good place to work.

  • Becoming disinterested in a job is a good reason to move from job to job as often as desired.

  • It is desirable to regularly move from job to job, looking for the job that best improves one’s lifestyle.

  • Repeatedly changing jobs is an ideal way to get a variety of job experiences.

  • People should be willing to change jobs as many times as necessary to get the best job possible.

Each candidate responded to at least 6 such statements as part of a 40 item personality test. These answers were coded 1 (less likely) to 5 (highly likely) and a measure on job-hopping likelihood was formed by averaging over all the questions. Figure 2 shows the distribution of job-hopping likelihood score () among all participants. This score formed the ground-truth for building the predictive model. The demographics of the candidates in terms of gender and the job role applied are shown in Table 1.

Figure 2: Job-hopping likelihood score distribution among all participants
Attribute Group Count
Gender Female 7,801
Male 9,242
Not specified 28,856
Job family Cabin crew 5,066
Call centre 1,587
Healthcare 16,305
Retail 14,241
Sales 7,445
Other 1,255
Table 1: Demographic breakdown of the participants in terms of gender and job family they applied to

3.2 Topic Analysis

We used LDA-based topic modelling as an exploratory tool to understand the relationship between terms and phrases used by candidates and their job-hopping likelihood. LDA assumes the presence of latent topics in a given set of text documents and attempts to probabilistically uncover these topics. See section 3.3.2 for a detailed description of LDA. Topics are generated over the vocabulary with each term having a particular affinity to each topic. These topic-term affinities can be utilized to comprehend a given topic as well as weights for terms in a word cloud to visualize the topic. We derived 100 such LDA topics for the whole text corpus.

Topic #29 showed the highest positive correlation of r=0.072 to job-hopping likelihood while highest negative correlation of r= -0.070 was shown by topic #49. Both correlations were significant at level. Figure 3 depicts the word cloud representation of the two topics.

Figure 3: Word clouds of significant LDA topics. (a) Topic #29: The highest positively correlated topic with job-hopping likelihood. (b) Topic #49: The highest negatively correlated topic with job-hopping likelihood.

On close analysis of the two word clouds, it can be observed that word cloud for topic #29 consists of terms indicating openness to new experience and opportunities such as think, new, idea, learn, looking, learning etc. In a further analysis we carried out (see section 4.1) it was found that the job-hopping intention is highly correlated with Openness to experience trait of HEXACO personality model which explains the above observation. On the other hand, topic #49, which is the highest negatively correlated topic, consists of terms such as safety, ensure, situation, process, procedure indicating a more process bound nature, the opposite of openness to novel experiences.

3.3 Text Representation

We evaluated four open-vocabulary approaches for representing textual information. Open-vocabulary approaches do not rely on a priori word or category judgments compared to traditional methods used by psychologists, such as self-reported surveys and questionnaires, that use predetermined sets of words to analyze (closed-vocabulary). With the recent advancements in Natural Language Processing (NLP), open-vocabulary approaches have gained popularity and shown better results

[35, 36, 39] over closed-vocabulary approaches such as LIWC [40]

used in the past for inferring personality from text. For comparison, we also trained a model using word categories in LIWC, the most commonly used lexicon for text analysis in the psychology domain.


Below we outline the five different text representation methods we used.

3.3.1 Tf-Idf

Term Frequency-Inverse Document Frequency (TF-IDF) [41] approach uses the relative frequency of occurrence of terms in the text corpus to model the language use. That is, the higher usage of a term in a response is scored high while offsetting for the number of responses the term occurs in. More formally, with , , and denoting term, response and the set of all responses respectively, , the number of times term appearing in response and , the number of responses where term appears,

(1)

where

(2)
(3)

We first tokenized the text responses from interview questions and developed a vectorized representation with the above TF-IDF scheme in n-dimensional space using unigrams, bigrams and trigrams of tokens. We experimented with n-dimensions=500, 1000 and 2000 of the most frequent n-grams (n=1,2,3) being used in the representation and found that n-dimensions=2000 to give the best outcomes.

3.3.2 Lda

Latent Dirichlet Allocation (LDA) [42] is a topic modelling approach that generates a given number of latent topics from a text corpus. LDA is a generative statistical model which assumes that a document (in our case a candidate response) relates to a number of latent topics while each latent topic is distributed across the vocabulary with different levels of affinities. Hence, a topic is usually described by the terms that have the highest affinities to that topic.

Using the notation defined in (1) and denoting a LDA topic,

(4)

We used the Gensim222https://radimrehurek.com/gensim/ software package for deriving 100 such topics. An example of a topic derived given by the terms with the highest affinities are {food, kitchen, restaurant, cleaning, chef, hospitality, worked, cooking, job}. It is important to note that the derivation of coherent topics such as the above is purely based on the statistical distributional properties of the terms in the text corpus.

3.3.3 Word Embeddings

The word embedding approaches [43, 44] to modelling language derives n-dimensional vectors to represent terms found in a given corpus, a numerical representation that preserves the contextual similarities between words. That is, similar words are placed closer to each other in the vector space. Hence word embeddings can be manipulated and made to perform tasks such as finding the degree of similarity between two words using intuitive arithmetic operations on the word vectors while retaining semantic analogies such as etc. Word embeddings based textual representations have been used in solutions that have achieved state-of-the-art results in many NLP tasks [45].

Word embeddings models are usually trained on large corpora such as Wikipedia or web pages gathered by a web crawler. We used the word embedding model available in the Spacy software package333https://spacy.io/, which is trained using the Glove algorithm [44] on content from common web crawl. To achieve a vector representation for a given response, we averaged across word embeddings of terms in that response.

3.3.4 Document Embeddings

The document embedding (also known as Doc2Vec) approach [46] to modelling text assigns n-dimensional vectors to variable-length textual content, such as sentences, paragraphs, and documents. While it is closely related to the Word2Vec method of word embeddings, the document vectors are intended to represent the concept of a document as opposed to the context of a word in Word2Vec [43]. Le and Mikilov [46] propose two Doc2Vec models, a distributed memory (Doc2Vec-DM) model and a distributed bag of words (Doc2Vec-DBOW) model. Doc2Vec-DM model is superior in terms of performance and usually achieves state-of-the-art results by itself. We used a Doc2Vec-DM model trained on content from Wikipedia to infer document vectors for candidate responses under this approach to modelling their language use.

3.3.5 Liwc

We also used the word categories from the Language Inquiry and Word Count (LIWC) lexicon [40], the most popular closed-vocabulary approach used in linguistic analysis and modelling in the social science domains, especially for assessing personality-related constructs. LIWC 2015 version consists of 76 categories and the frequencies of occurrence of words in these categories in each candidate response normalized by the response length are used as features in modelling the language use.

Using the notation defined in (1) and denoting a category in LIWC lexicon, category frequency,

(5)

3.4 Language to Job-hopping Likelihood Model Building

The above representations were used to build a regression model with the Random Forest [47] algorithm using the corresponding job-hopping likelihood scores as the target. We selected Random Forest for its known superior performance compared to many other regression algorithms. We see as future work to compare the outcomes using different algorithms. We find it as sufficient to show the outcomes on a single algorithm in order to establish the correlation between language use and job-hopping likelihood as any improvement made over our findings using a different regression algorithm would only make the case for language-based inference of job-hopping likelihood stronger.

We used 80% of the data to train the model while the rest of the data was used to validate the accuracy of the trained model. We experimented with different minimum text response lengths, excluding records for candidates with responses shorter than the selected minimum word length. The hypothesis behind this exercise was that responses that are too short might not have enough textual content to predict the candidates’ job-hopping likelihood. We strived to find a balance between the minimum text response length and the data available for training the model to train the best predictive model. Table 2 presents the number of records for different minimum response lengths.

Min. response length (words) Number of records (N)
50 45,899
100 32,472
150 23,675
200 18,210
Table 2: Data size for different minimum response length restrictions

4 Results and Discussion

We evaluated the trained ‘language use to job-hopping likelihood’ regression models on the remaining 20% of the data. The models are evaluated on the correlation coefficient between the actual job-hopping likelihood score and score produced by the trained model. The correlation coefficient (r) is a statistical measure of the strength of the relationship between two interdependent variables. In this case, it is the strength of the relationship between the actual and the inferred job-hopping likelihood scores. Figure 4 shows the accuracies in terms of correlation across different minimum response lengths and language modelling approaches.

Figure 4: Model accuracy vs. minimum text length in words

Language use representation using Glove word embeddings with minimum response length of 150 words achieved the highest correlation of r=0.35. It is important to note that six of the correlations fell above r=0.3, typically considered as a correlational upper-limit in personality research when predicting behaviour [48]. It is also important to note that apart from the correlations for LIWC with minimum text length of 50, 100 and 150 all other correlations had a . These results indicate that all open-vocabulary approaches across all minimum lengths considered recorded significant correlations, demonstrating that the language one uses in responding to typical interview questions are predictive of one’s likelihood towards job-hopping.

Overall, word embeddings based models recorded the highest corrections across all minimum lengths analysed. This is specifically important given that word embeddings based models are more generalizable to unseen words compared to models based on TF-IDF and LDA, which are limited to the vocabulary seen in the training corpus. The generalizability comes from the content used in training the word embedding model; In our case, the word embeddings used were trained on very generic content from web pages crawled from the Internet. Compared to the superior results achieved by word embedding based models, document embedding based models fell short of in terms of the accuracy. This, we believe, is due to the nature of the content used to train the document embedding model. We used content from Wikipedia to train our document embedding model and differences in actual content and writing style between Wikipedia and candidate responses may have contributed to the degraded performance of the document embedding based models. Further research is required to validate this and Doc2Vec models trained on other content such as tweets are options to consider.

Overall, minimum response length of 50 words achieved the weakest results confirming our hypothesis that responses that are too short might not have enough textual content to predict the candidates’ job-hopping likelihood. However, none of the models showed an increase in accuracy but a decrease or maintaining the same accuracy (with the exception of LIWC based model), when moving to a minimum length of 200 words. Given LIWC depends on counts of words related to pre-defined categories, we assume that more words in a response raise the possibility of finding more LIWC classified words in the response. However, the overall poor performance of LIWC highlights the limitations of closed-vocabulary approaches where a tediously developed lexicon for a certain domain cannot be applied to a different domain or generic content and vice versa.

Following sections describe some of the further analysis we performed on the best ‘language use to job-hopping likelihood’ model (min. response length=150, using word embeddings features) to get a deeper understanding of the model’s behaviour. These analyses were carried out on the remaining 20% of the data that were left out of training the regression model. The gender and role family composition of the this test data set can be found Tables 4 and 5.

4.1 Correlations with Personality and Language Characteristics

We evaluated the correlations between the output of the trained model (i.e. the inferred job-hopping likelihood) and candidates’ personality measured in terms of the six-factor HEXACO trait model [38]. HEXACO, a six-factor model of personality introduced by Ashton and Lee [38, 49] is closely related to the Big Five model [50] of personality but proposed as a better alternative, especially in explaining work-related behaviours. The six factors are Honesty-humility (H), Emotionality (E), eXtraversion (X), Agreeableness (A), Conscientiousness (C) and Openness to experience (O). We calculated each candidates HEXACO trait values using a language to HEXACO inference model described in [18].

Variable Correlation coefficient
HEXACO personality traits
        Honesty-humility -0.02
        Emotionality 0.06
        Extraversion 0.01
        Agreeableness -0.09
        Conscientiousness -0.00
        Openness to experience 0.25
Response length in words -0.15
Sentence count -0.14

Formality score (F-score)

-0.22
Coleman Liau index -0.16
Number of unique difficult words -0.13
Table 3: Correlations with the inferred job-hopping likelihood score

Table 3 presents the correlations between the inferred job-hopping likelihood score and HEXACO personality traits, response length in words and number of sentences, Formality score (F-score) - a measure of formality and contextuality proposed by Heylighen and Dewaele [51], Coleman Liau index - a commonly used measure of readability [52] and count of “difficult words”, identified by Dale and Chall [53] as words not included in a list of 3000 “easy words” commonly found in written English. F-score, Coleman Liau index and difficult words are measures of language proficiency and readability.

The negative correlations with response length and especially F-score indicate that candidates who are likely to hop jobs wrote less compared to others and used less sophisticated language. Moreover, the positive correlation of r=0.25 with the HEXACO Openness to experience trait indicates candidates who are open to experiences are more likely to hop jobs. The positive correlation between Openness to experience and turnover confirms what has been observed by Sarwar et al. [29], Zimmerman [9], Timmerman [8] and Anderson et al. [30]. It is also interesting to note that the highest negative correlation (r=-0.09) with a HEXACO trait was recorded with Agreeableness, a personality trait related to leniency in judging others, more willing to compromise and cooperate with others, and can easily control their temper [38]. The results indicate that personalities low in Agreeableness are more likely to hop jobs.

4.2 Relationship with Candidate Demographics

We inspected the relationship between the inferred job-hopping likelihood score and the job family the candidate applied to and their gender.

Job family Count Mean
Cabin crew 1,008 2.25
Call centre (outbound) 207 2.38
Healthcare 1,435 2.29
Retail 852 2.37
Sales 1,037 2.39
Other 195 2.27
Table 4: Inferred job-hopping likelihood statistics for each job family

Table 4 presents the statistics for each job family. The mean values for job families suggest that candidates for job families call centre (outbound), retail and sales have a higher tendency to hop jobs, which is in line with the general understanding of the job roles. Most of these are casual roles where candidate mobility is high, especially in outbound call centres. Sales roles are known to be stress causing related to target-driven nature of the operations. Moreover, an ANOVA analysis of job family data suggests that mean values demonstrate a statistically significant difference across job families.

Table 5 presents the statistics for gender. While the mean value for males is slightly higher than females’, the effect size is 0.15 suggesting the difference is not significant. This is an important indication towards the trained model not showing bias towards any gender.

Gender Count Mean
Female 1,339 2.31
Male 1,348 2.33
Not specified 2,047 2.32
Table 5: Inferred job-hopping likelihood statistics for gender

5 Conclusion and Future Work

Frequent movement from job to job or “job-hopping” as its commonly known is found to be associated with one’s personality. The ability to identify a job applicant’s likelihood for job-hopping can help organizations avoid future costs associated with voluntary turnover due to job-hopping. In this paper, we presented a novel approach to predicting job-hopping likelihood using answers to typical interview questions related to past behaviour and situational judgement. Using data from over 45,000 individuals who answered open-ended interview questions and self-rated themselves on a job-hopping motive scale, we built a regression model for inferring job-hopping likelihood. We compared the performance of four open-vocabulary text representation methods (namely terms, topics, word embeddings and document embeddings) and one closed-vocabulary method (LIWC). The Glove word embedding based model achieved the highest correlation of r=0.35 () between interview response text and job-hopping likelihood. All other open-vocabulary representations achieved correlations above 0.25 (), highlighting a statistically significant positive correlation between interview responses and job-hopping likelihood. We further demonstrated that ones job-hopping likelihood is positively correlated (r=0.25) with the trait Openness to experience and negatively correlated with Agreeableness (r=-0.09) as found in the six-factor HEXACO personality model. In other words, the more open someone is for new experiences and less lenient with views of others, the more likely he/she will show job-hopping behaviour.

We find the above outcome to be significant in at least two ways. It removes the dependency on resume based job histories as a source for inferring a job-applicants tendency to job hop. Resumes are known to induce bias in the hiring process and especially ineffective with newcomers to the job market with no prior significant job history. Secondly, the ability to infer job-hopping likelihood computationally from interview responses uplifts the utility of the interview as a multi-measure assessment that can be conducted digitally (e.g.text chat) at scale and cost-effectively giving every candidate an opportunity to express themselves. Interview as an assessment is preferred by applicants over traditional assessments such as personality tests.

Further work is required in assessing the predictive validity of the outcome, i.e. establishing the correlation between inferred job-hopping likelihood and actual job-hopping behaviour. This requires a longitudinal study or following the career journey of an applicant sample. In our current study, we used only the semantic level features (terms, topics etc). Exploring whether other types of features can further increase the accuracy, is another useful future extension. These can include the use of parts of speech (POS), use of emojis and multi-modal information such as audio and video signals captured while candidates answer the questions. Exploring the performance of other available regression algorithms, including neural network approaches, and using more advanced language representations such as BERT

[54], may help increase the accuracy of the regression model further.

References

  • [1] T. Lam, A. Lo, and J. Chan, “New employees’ turnover intentions and organizational commitment in the hong kong hotel industry,” Journal of Hospitality & Tourism Research, vol. 26, no. 3, pp. 217–234, 2002. [Online]. Available: https://doi.org/10.1177/1096348002026003002
  • [2] J. Chatzky, “Job-hopping is on the rise. should you consider switching roles to make more money?” NBCNews, 2018. [Online]. Available: https://www.nbcnews.com/better/business/job-hopping-rise-should-you-consider-switching-roles-make-more-ncna868641
  • [3] C. J. Lake, S. Highhouse, and A. G. Shrift, “Validation of the job-hopping motives scale,” Journal of Career Assessment, vol. 26, no. 3, pp. 531–548, 2018. [Online]. Available: https://doi.org/10.1177/1069072717722765
  • [4] E. Ghiselli, “Some perspectives for industrial psychology,” American Psychologist, vol. 29, pp. 80–87, 01 1974.
  • [5] K. Barton and R. B. Cattell, “Personality factors related to job promotion and turnover,” Journal of Counseling Psychology, vol. 19, no. 5, pp. 430–435, 1972.
  • [6] J. M. Jenkins, “Self-monitoring and turnover: The impact of personality on intent to leave,” Journal of Organizational Behavior, vol. 14, no. 1, pp. 83–91, 1993. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/job.4030140108
  • [7] J. F. Salgado, “The big five personality dimensions and counterproductive behaviors,” International Journal of Selection and Assessment, vol. 10, no. 1, pp. 117–125, 2002. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/1468-2389.00198
  • [8] T. A. Timmerman, “Predicting turnover with broad and narrow personality traits,” International Journal of Selection and Assessment, vol. 14, no. 4, pp. 392–399, 2006. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-2389.2006.00361.x
  • [9] R. D. Zimmerman, “Understanding the impact of personality traits on individuals’ turnover decisions: A meta-analytic path model,” Personnel Psychology, vol. 61, no. 2, pp. 309–348, 2008. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1744-6570.2008.00115.x
  • [10] M. Bertrand and S. Mullainathan, “Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination,” National Bureau of Economic Research, Working Paper 9873, July 2003. [Online]. Available: http://www.nber.org/papers/w9873
  • [11] S. K. Kang, K. A. DeCelles, A. Tilcsik, and S. Jun, “Whitened résumés: Race and self-presentation in the labor market,” Administrative Science Quarterly, vol. 61, no. 3, pp. 469–502, 2016. [Online]. Available: https://doi.org/10.1177/0001839216639577
  • [12] J. Golbeck, C. Robles, M. Edmondson, and K. Turner, “Predicting personality from twitter,” in 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, 2011, pp. 149–156.
  • [13] C. Sumner, A. Byers, R. Boochever, and G. J. Park, “Predicting dark triad personality traits from twitter usage and a linguistic analysis of tweets,” in 2012 11th International Conference on Machine Learning and Applications, vol. 2, 2012, pp. 386–393.
  • [14] D. Xue, Z. Hong, S. Guo, L. Gao, L. Wu, J. Zheng, and N. Zhao, “Personality recognition on social media with label distribution learning,” IEEE Access, vol. 5, pp. 13 478–13 488, 2017.
  • [15] M. M. Tadesse, H. Lin, B. Xu, and L. Yang, “Personality predictions based on user behavior on the facebook social media platform,” IEEE Access, vol. 6, pp. 61 959–61 969, 2018.
  • [16] Z. Wang, C. Wu, K. Zheng, X. Niu, and X. Wang, “SMOTETomek-based resampling for personality recognition,” IEEE Access, vol. 7, pp. 129 678–129 689, 2019.
  • [17] F. Iacobelli, A. J. Gill, S. Nowson, and J. Oberlander, “Large scale personality classification of bloggers,” in Affective Computing and Intelligent Interaction, ser. Lecture Notes in Computer Science, S. D’Mello, A. Graesser, B. Schuller, and J.-C. Martin, Eds.     Springer, 2011, pp. 568–577.
  • [18] M. Jayaratne and B. Jayatilleke, “Predicting personality using answers to open-ended interview questions,” IEEE Access, vol. 8, pp. 115 345–115 355, 2020.
  • [19] Y. Neuman and Y. Cohen, “A vectorial semantics approach to personality assessment,” Nature Scientific Reports, vol. 4, no. 1, pp. 1–6, 2014. [Online]. Available: https://www.nature.com/articles/srep04761
  • [20] J. Levashina, C. J. Hartwell, F. P. Morgeson, and M. A. Campion, “The structured employment interview: Narrative and quantitative review of the research literature,” Personnel Psychology, vol. 67, no. 1, pp. 241–293, 2014. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/peps.12052
  • [21] M. Mcdaniel, D. Whetzel, F. Schmidt, and S. Maurer, “The validity of employment interviews: A comprehensive review and meta-analysis. the journal of applied psychology, 79, 599-616,” Journal of Applied Psychology, vol. 79, pp. 599–616, 08 1994.
  • [22] P. Begley and L. Dunne, “Turnover and retention research report,” 2018. [Online]. Available: https://www.ahri.com.au/media/1222/turnover-and-retention-report_final.pdf
  • [23] S. Cho, R. H. Woods, S. (Shawn) Jang, and M. Erdem, “Measuring the impact of human resource management practices on hospitality firms’ performances,” International Journal of Hospitality Management, vol. 25, no. 2, pp. 262–277, 2006. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0278431905000459
  • [24] T. Mahan, D. Nelms, and C. Bearden, “Retention report: Truth & trends in turnover,” 2018. [Online]. Available: http://info.workinstitute.com/2018retentionreport
  • [25] T.-Y. Park and J. D. Shaw, “Turnover rates and organizational performance: A meta-analysis,” Journal of Applied Psychology, vol. 98, no. 2, pp. 268–309, 2013.
  • [26] W. S. Harman, T. W. Lee, T. R. Mitchell, W. Felps, and B. P. Owens, “The psychology of voluntary employee turnover,” Current Directions in Psychological Science, vol. 16, no. 1, pp. 51–54, 2007. [Online]. Available: https://doi.org/10.1111/j.1467-8721.2007.00474.x
  • [27] V. Ariyabuddhiphongs and S. Marican, “Big five personality traits and turnover intention among thai hotel employees,” International Journal of Hospitality & Tourism Administration, vol. 16, no. 4, pp. 355–374, 2015. [Online]. Available: https://doi.org/10.1080/15256480.2015.1090257
  • [28] L. C. Hong and S. Kaur, “A relationship between organizational climate, employee personality and intention to leave,” International Review of Business Research Papers, vol. 4, no. 3, pp. 1–10, 2008.
  • [29] A. Sarwar, S. Hameed, and H. Aftab, “Study to explore the impact of personality traits on employee turnover in public and private sector,” Middle-East Journal of Scientific Research, vol. 16, no. 9, pp. 1249–1254, 2013.
  • [30] S. E. Woo, “A study of ghiselli’s hobo syndrome,” Journal of Vocational Behavior, vol. 79, pp. 461–469, 10 2011.
  • [31] N. Anderson, J. Salgado, and U. Hülsheger, “Applicant reactions in selection: Comprehensive meta-analysis into reaction generalization versus situational specificity,” International Journal of Selection and Assessment, vol. 19, pp. 291–304, 08 2010.
  • [32] J. P. Hausknecht, D. V. Day, and S. C. Thomas, “Applicant reactions to selection procedures: An updated model and meta-analysis,” Personnel Psychology, vol. 57, no. 3, pp. 639–683, 2004. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1744-6570.2004.00003.x
  • [33] J. W. Pennebaker and L. A. King, “Linguistic styles: Language use as an individual difference.” Journal of personality and social psychology, vol. 77, no. 6, p. 1296, 1999.
  • [34] L. A. Fast and D. C. Funder, “Personality as manifest in word use: correlations with self-report, acquaintance report, and behavior.” Journal of personality and social psychology, vol. 94, no. 2, pp. 334–46, 2008.
  • [35] A. J. Gill, S. Nowson, and J. Oberlander, “What are they blogging about? personality, topic and motivation in blogs,” in Third International AAAI Conference on Weblogs and Social Media, 2009. [Online]. Available: https://www.aaai.org/ocs/index.php/ICWSM/09/paper/view/199
  • [36] J. B. Hirsh and J. B. Peterson, “Personality and language use in self-narratives,” Journal of Research in Personality, vol. 43, no. 3, pp. 524–527, 2009. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0092656609000439
  • [37] L. Qiu, H. Lin, J. Ramsay, and F. Yang, “You are what you tweet: Personality expression and perception on twitter,” Journal of Research in Personality, vol. 46, no. 6, pp. 710–718, 2012. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S009265661200133X
  • [38] M. C. Ashton and K. Lee, “Empirical, theoretical, and practical advantages of the HEXACO model of personality structure:,” Personality and Social Psychology Review, 2007. [Online]. Available: https://journals.sagepub.com/doi/10.1177/1088868306294907
  • [39] H. A. Schwartz, J. C. Eichstaedt, M. L. Kern, L. Dziurzynski, S. M. Ramones, M. Agrawal, A. Shah, M. Kosinski, D. Stillwell, M. E. P. Seligman, and L. H. Ungar, “Personality, gender, and age in the language of social media: The open-vocabulary approach,” PLOS ONE, vol. 8, no. 9, p. e73791, 2013. [Online]. Available: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0073791
  • [40] J. W. Pennebaker, R. L. Boyd, K. Jordan, and K. Blackburn, “The development and psychometric properties of LIWC2015,” 2015, accepted: 2015-09-16T13:00:41Z. [Online]. Available: https://repositories.lib.utexas.edu/handle/2152/31333
  • [41] M. Christopher, R. Prabhakar, and S. Hinrich, Introduction to Information Retrieval.     Cambridge University Press, 2008.
  • [42] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” Journal of Machine Learning Research, vol. 3, pp. 993–1022, 2003. [Online]. Available: http://www.jmlr.org/papers/v3/blei03a
  • [43]

    T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in

    Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS’13.     Curran Associates Inc., 2013, pp. 3111–3119.
  • [44] Jeffrey Pennington, R. Socher, and Christopher Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532–1543. [Online]. Available: https://www.aclweb.org/anthology/D14-1162
  • [45] J. Camacho-Collados and M. T. Pilehvar, “From word to sense embeddings: A survey on vector representations of meaning,” CoRR, vol. abs/1805.04032, 2018. [Online]. Available: http://arxiv.org/abs/1805.04032
  • [46] Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ser. ICML’14.     JMLR.org, 2014, pp. II–1188–II–1196.
  • [47] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001. [Online]. Available: https://doi.org/10.1023/A:1010933404324
  • [48] B. W. Roberts, N. R. Kuncel, R. Shiner, A. Caspi, and L. R. Goldberg, “The power of personality: The comparative validity of personality traits, socioeconomic status, and cognitive ability for predicting important life outcomes,” Perspectives on Psychological Science, vol. 2, no. 4, pp. 313–345, 2007. [Online]. Available: https://doi.org/10.1111/j.1745-6916.2007.00047.x
  • [49] K. Lee and M. C. Ashton, The H Factor of Personality: Why Some People are Manipulative, Self-Entitled, Materialistic, and Exploitive—And Why It Matters for Everyone.     Wilfrid Laurier Univ. Press, 2013.
  • [50] L. R. Goldberg, “The structure of phenotypic personality traits.” American psychologist, vol. 48, no. 1, p. 26, 1993.
  • [51] F. Heylighen and J.-M. Dewaele, “Variation in the contextuality of language: An empirical measure,” Foundations of Science, vol. 7, no. 3, pp. 293–340, 2002.
  • [52] M. Coleman and T. L. Liau, “A computer readability formula designed for machine scoring,” Journal of Applied Psychology, vol. 60, no. 2, pp. 283–284, 1975.
  • [53] E. Dale and J. S. Chall, “A formula for predicting readability: Instructions,” Educational Research Bulletin, vol. 27, no. 2, pp. 37–54, 1948, publisher: Taylor & Francis, Ltd. [Online]. Available: https://www.jstor.org/stable/1473669
  • [54] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805