affective visual journal using deep neural networks for NLP and image synthesis (Python, Tensorflow, Keras, NumPy, Scikit-Learn)
We present Lemotif. Lemotif generates a motif for your emotional life. You tell Lemotif a little bit about your day -- what were salient events or aspects and how they made you feel. Lemotif will generate a lemotif -- a creative abstract visual depiction of your emotions and their sources. Over time, Lemotif can create visual motifs to capture a summary of your emotional states over arbitrary periods of time -- making patterns in your emotions and their sources apparent, presenting opportunities to take actions, and measure their effectiveness. The underlying principles in Lemotif are that the lemotif should (1) separate out the sources of the emotions, (2) depict these sources visually, (3) depict the emotions visually, and (4) have a creative aspect to them. We verify via human studies that each of these factors contributes to the proposed lemotifs being favored over corresponding baselines.READ FULL TEXT VIEW PDF
affective visual journal using deep neural networks for NLP and image synthesis (Python, Tensorflow, Keras, NumPy, Scikit-Learn)
Our emotional well being is important to us. In part due to its subjective, and well, emotional nature, it is difficult to objectively find patterns in what we feel, how often we feel it, and what the source of those feelings tends to be. Without this assessment, it is difficult to tweak our choices to optimize for our emotional well being.
Numerous tools exist for logging one’s emotional state. But data entry is dry. Can we add a bit of creativity to making logging more engaging? It would be nice if there was something one could look forward to after entering each log; something fun, something visual, something one could potentially share with their friends and family.
Meet Lemotif. Lemotif creates an abstract visual depiction – a motif – of your emotional states over a period of time. We call this abstract visual depiction a lemotif.111Coincidentally related to, but distinct from Leitmotif, which is a ““short, constantly recurring musical phrase” associated with a particular person, place, or idea.” (Wikipedia). We call our system Lemotif, and the creative visual abstraction it generates a lemotif. The idea behind Lemotif is the following. You tell Lemotif a little bit about your day – what were salient events or aspects and how they made you feel. Lemotif will generate visual motifs that are an abstract depiction of your emotions and their sources. Over time, Lemotif will create visual motifs to capture a summary of your emotional states over arbitrary periods of time – making patterns in your emotions and their sources apparent, presenting opportunities to take actions, and measure their effectiveness.
The concrete instantiation of Lemotif that we present in this paper is shown in Fig. 1. Details of the approach are in the Approach section. The core principles behind our approach are that: (1) The lemotif should separately depict each salient topic of the day so as to make the source of feelings apparent to the user. This is especially important when generating lemotifs for periods of time, so users can take actions to make changes in their life towards improved emotional health222Of course, emotional health is a serious and complex issue. But even simple tools can often make a non-trivial difference., if they so desire. (2) The lemotif should depict the topic visually so the feeling-topic association is more apparent and better grounded in the user’s mind. This also adds an element of creativity and interestingness to the lemotif, which can make it more engaging. Lemotif depicts the topics visually via the icons or shapes seen in Fig. 1. (3) In similar spirit, the lemotif should depict the feelings visually. Lemotif depicts the feelings visually via the colors seen in Fig. 1. (4) The lemotif should be creative so as to be more engaging to a user, making them more likely to journal regularly, and look for patterns to improve their emotional well being. This also presents opportunities for users to share their lemotif with friends and family and/or on social media, potentially resulting in connections with people and again, improved emotional health. The circle packing design shown in Fig. 1 plays this role.
We evaluate Lemotif via human studies that check whether the topic-icon and feeling-color mappings are meaningful to users, whether the four factors listed above make the lemotif more favorable to subjects compared to associated baselines, whether subjects say they would use (and even pay for!) a system like Lemotif, whether subjects believe the lemotif makes their “creative juices flow”, etc. We report favorable results on all these fronts.
To the best of our knowledge, there is limited work in generating visual depictions of journal entries. In the next paragraph we describe some existing journaling tools. We first discuss a few directions that are relevant to our work at a high level. Our work generates a visual depiction of a textual journal entry. Approaches in multi-modal AI that generate natural images from their language descriptions [Reed et al.2016] are relevant. Our work focuses on emotions and depicts them visually. Relatedly, there is fascinating work on projecting a spectrum of human emotions on an interactive map with associated video clips that elicit those emotions [Cowen and Keltner2017] and to audio gasps that people make when expressing these emotions [Cowen et al.2018]
. Such efforts provide a fertile ground for studying the multimodal nature of human emotions. Our motivation in generating visual depictions of a user’s journal entry is to make the connection between a user’s emotions and their sources more apparent. This motivation of making patterns in data more apparent is the basis of all data visualization techniques. Doodles are a way to reinforce the meaning or theme of a piece of text. Emojis are frequently visual depictions of emotions. It would be interesting to generate visual depictions of journal entries by blending emojis[Martins, Cunha, and Machado2018] that depict the emotions reported by users along with emojis coming from a theme that align with the topic mentioned by the users. [Salevati and DiPaola2015] studied the emotions evoked in viewers of paintings generated by a creative AI system. Their findings could inform our lemotif designs.
Most existing journaling tools and apps are meant to be a way for users to log their lives, and not necessarily to enable extraction of actionable patterns in the logs. Emphasis is often on easy incorporation of multimedia, hand annotations, maps, search tools, etc. Our focus is more on making associations between a user’s feelings and aspects of their life apparent. When journaling apps claim to be ‘visual’ (e.g., HeyDay), they typically refer to allowing visual input modalities (e.g., images, videos). Our work produces a visual modality as an output. Life Calendar comes closest to our approach, where it shows a single-colored dot (red, yellow, or green) for each week that captures the mood of the user in that week (negative, neutral, positive). This allows one to find correlations between time of month or year and emotion (e.g., happier in the summer). But it does not help identify sources of emotions on a day-to-day basis, which may be more actionable. In our experiments, we compare to a visualization that mimics this and find that subjects strongly prefer our nuanced and creative visualization.
Icons used to represent various topics. We select icons from the Noun Project (top) and process them to binarize, resize and recenter before extracting the outer shape (bottom).
Our current instantiation of Lemotif works as follows.
The user is prompted to talk about (up to) three salient aspects of their day. The user indicates these aspects from a pre-defined list of 11 topics (e.g., work, family). These can be indicated via #s (that auto-complete from the pre-defined list), or through a drop down menu, or a list of radio buttons. We use a list of radio buttons in our experiments for ease of implementation. A better UI is part of future work. The user also indicates how these aspects made them feel by selecting up to 4 feelings from a pre-defined list of 18 feelings (e.g., happy, anxious). The limit of 3 topics and 4 feelings is simply so the generated lemotif is easy to comprehend. These limits can be customized by the user. The user then describes that aspect of their day and their feelings in free-form text. This free-form text is currently not used by Lemotif. Note that if the UI supports #s with auto-complete, the topic and feeling need not be separately identified. They can be integrated into the free-form text (as shown in Fig. 1) and trivially extracted.
The 11 topics in our pre-defined list are shown in Fig. 2. This list was determined by a mix of brainstorming and searching online for what topics users typically talk about in their journals. As part of our evaluation, we asked users if they felt a topic they would like to talk about was missing. 99 subjects out of 100 said this list was sufficient. 1 user suggested adding pets as a topic.
The 18 feelings in our pre-defined list are shown in Fig. 3. This list was curated primarily from [Cowen and Keltner2017] and our assessment of what emotions are likely on a day-to-day basis. Again, as part of our evaluation, we asked users if they felt a feeling they would like to talk about was missing. All 100 subjects said the list was sufficient. A couple of quotes from subjects: “These are some of the more important events and feelings that occur daily. The list really sums up what happens in my daily life.” and “Everything pertinent was covered above.”
Lemotif has a pre-defined mapping from topics to visual icons that depict that topic. These are shown in Fig. 2. These icons or shapes could also be specified by the user. To identify our list of icons, we started with the Noun Project333https://thenounproject.com/. The Noun Project has over two million binary icons created by designers all over the world. We searched the Noun Project for each of the topics to ensure that the icons we pick are relevant to the topic (e.g., book for school). From the relevant icons, we selected those that are not visually complex so the lemotif is clear. Simple icons also allow for more freedom in the kinds of creative visualizations one could explore. Intricacies of an icon will likely not be legible for a variety of creative visualizations. We automatically binarize the image, crop the icon, and resize it to a canonical size. To further simplify the icons, we post-process them to retain only their outer shape, and discard the inner details. This was done by keeping only the extreme points of the shape in each row and column of the image. This provides a thin and sparse outline of the icon. This might suffice for many visualization techniques, but for completeness, we dilate the sparse outline using morphological filtering. The resulting icons are shown in the bottom row of Fig. 2. In our experiments we evaluate how natural this topic-icon mapping is for subjects.
Lemotif also has a pre-defined mapping from feelings to colors that depict that feeling. See Fig. 3. These colors were selected such that they are commonly associated with the corresponding feelings (e.g., dark red for angry) as indicated by studies [Nijdam2005] and online resources, and are distinct444https://sashat.me/2017/01/11/list-of-20-simple-distinct-colors/ so as to not be confused with each other. These can also be customized by the user. In our experiments we evaluate how natural this feeling-color mapping is for subjects. We experimented with [Kawakami et al.2016], a neural model that maps any word to a color. We found that while it is effective for nouns associated with objects that have canonical colors (e.g., eggplants are purple), it provided similar colors for different emotion words.
Based on the topics mentioned by the user in their entry, the associated feelings, and the mappings described above, Lemotif generates a creative abstract visual depiction of the user’s day. Feelings are amorphous and so we did not want the visual depiction to be overly structured. We wanted the shape of the icon that represents the topic to be legible, but without the boundary being overly emphasized. Same for the colors associated with the feelings. We wanted them to be interspersed in the shape and not ‘boxed in’. We experimented with two visualizations that we call circle packing (Fig. 4 A1) and string doll (Fig. 4 A2).
Circle packing is a well-studied problem in mathematics: how do we pack a space with as many non-overlapping circles as possible? A class of algorithmic art approaches have been motivated by this555http://jdobr.es/blog/algorithmic-art-circle-pack https://generativeartistry.com/tutorials/circle-packing , and with the introduction of randomness, color, and other design choices, created spectacular patterns. We follow similar ideas for our circle packing visualization. Recall that given a topic (and its associated icon), and a list of feelings reported by the user (and their associated colors), our goal is to generate a visualization of that icon in those colors.
In circle packing, we pack the icon with circles of a certain distribution of sizes (pre-defined but customizable), uniformly randomly colored by one of the colors. We start with a set of circle radii and the desired number of circles to be placed in the icon for each radii. Starting from the largest size, we sample a random location in the icon. If a circle can be placed there without any part of the circle falling outside the icon, we place the circle. We then sample another random location in the icon. If a circle can be placed there without any part of the circle falling outside the icon and without it overlapping the existing circle, we place the circle there. If not, we sample another random location and try again till a max number of trials is reached (or the circle is successfully placed). This is repeated for the specified number of circles to be placed of that size. We then move on to the smaller of the circle sizes and repeat the process.
In our visualization, for a canvas of about 2225 2225 pixels where the icon is placed in the center, we chose circle radii to be 180, 90, 45 and 15 pixels, and placed 12, 24, 48, and 192 circles of those sizes respectively. To add some texture, we add an outline around each circle. If the hue of the color of the circle is more than 0.5 (i.e., it is a bright color), the color of the outline is half way between the color of the circle and a significantly darker version of it (fairly close to black). Similarly, if the hue of the color is less than 0.5, the color of the outline is set to be half way between the color and a significantly lighter version. An example circle packing visualization can be see in Fig. 4 A1.
In this visualization, we render an icon by drawing strokes that connect two random points on the icon’s boundary, without the stroke going outside the boundary of the icon. The strokes are colored uniformly randomly by one of the colors corresponding to the feelings the user reported for that topic. These strokes are quadratic bezier curves with end points being two random points on the icon boundary. As the control point, we take the mid point of the two end points and add zero-mean gaussian noise to it. The standard deviation of the gaussian is set to 20% of the size of the canvas. So for a 650650 canvas, the noise added to each co-ordinate of the mid-point is sampled from . The width of the stroke is sampled from a distribution. th of the strokes are 4 pixels wide, th are 5 pixels wide, and the remaining th are 6 pixels wide. We add a total of 100 strokes to each icon. To add some texture to the visualization, each stroke is overlaid by a stroke that is a quarter of its width. The color of the overlaid stroke is picked in the same way as the color of the outline in the previous circle packing approach. An example string doll visualization can be see in Fig. 4 A2.
The final lemotif is a concatenation of the visualizations for each topic that was salient in the user’s day (Fig. 4 A1, A2). If the user reports more salient topics, the lemotif would have more elements in it. We believe one-to-three topics is the appropriate amount of information a user can process visually, but this can be customized by the user. Similarly, the number of feelings the user reports for a topic dictates the number of colors used to create the corresponding visual. We believe more than 4 colors would be overwhelming, but can be customized by the user.
The discussion above (and in the rest of the paper for the most part) talks about generating a lemotif
for a day. The same approach can be used over arbitrary periods of times (e.g., a week, a month, a year) by identifying frequently occurring salient topics and associated feelings. A heuristic we currently use is that given a period of time, we pick the topics that have been mentioned more than a third of the time, and feelings that have been mentioned more than a third of a time for that topic.
A lot of room remains for exploring other creative visual depictions, including allowing users to design these. Future work also includes incorporating the free-form text in the journal entry while generating the lemotif
, and ideally extracting the salient topics and feelings from the free-form text directly using natural language processing, freeing the user from having to explicitly specify them. One could also explore directions where the user is asked for more fine-grained information about each feeling (e.g., on a scale of 1-5), which is then incorporated in the visual. We hypothesize that such fine-grained information would be difficult to glean from the visual, and would not be worth the added effort from the user.
We collected 100 journal entries from 100 (anonymous) subjects on Amazon Mechanical Turk (AMT) describing their day. We could not control for the time of day when a subject did our task, so we asked them to talk about “yesterday” (instead of “today” which is common for journals). As described earlier, each entry included up to 3 topics (from the list of 11 topics described earlier). Each topic mentioned up to 4 feelings (from the list of 18 feelings described earlier), along with free-form text describing that aspect of their day. 666Example entry: Family: calm, happy, satisfied. My husband and I continued playing through Horizon Zero Dawn together. The game is turning out to be really great. It’s long so it’s a little annoying when he’s ready to play and I’m not yet but overall it’s a really fun activity that we’re doing and appreciating together. It gives me many warm fuzzy feelings that I can share an appreciation for art and story with him like that. Work: bored, frustrated. Work sucked so bad yesterday. I wasn’t feeling well because I’ve been sick so it was hard to concentrate. Or rather my brain wanted to concentrate more on how I my [sic] throat hurt rather than on the work I was supposed to be doing. Food: excited, happy, proud. I baked lemon bars late the night before and finally got to try them yesterday and they were delicious! They were super rich and I have to cut the pieces smaller but I was very proud of myself that I worked up the energy to bake them and that I did a good job. Note that because subjects were not writing entries in a “real” journal, they may have provided us with an entry that is not honest.777Of course, there is another level of “noise” in that even if the description is honest, it is still what the user believes they are feeling, and may not be what they are actually feeling. This is a concern for all journaling tools as well as scientific studies like [Cowen and Keltner2017] that maps out the entire spectrum of human emotions based on self-reported emotions. Our goal is this work is to generate a visual summary of the entries (and to evaluate depictions). The authenticity of the entry is less crucial towards this goal. Subjects were from the US (to ensure fluent English), had 95% approval rating, and had completed at least 5000 HITs in the past. The same qualifications were also used for all evaluation tasks discussed later in the paper.
To get a sense for the data we collected, we ran some analysis. On average, an entry was 507 characters (92 words) long; subjects reported 2.8 topics per day, and 1.9 feelings per topic per day. Fig. 5 shows the distribution of topics and feelings subjects chose to talk about. Family, food, and work were frequently talked about, while school was not talked about much at all (potentially because only a fifth [Levay, Freese, and Druckman2016] to a third [Ross et al.2009]
of AMT workers are estimated to be students). Being happy and satisfied is reported most frequently, while being afraid, ashamed, awkward are rarely mentioned. No one reported being jealous. Of the negative feelings, subjects reported being frustrated and anxious most often. Note that, as mentioned earlier, it is likely that some of the entries are not authentic. This analysis is not meant to be a statement about how people in general or even AMT subjects specifically actually feel. It is as an analysis of the data we have collected, that we will generatelemotifs for and evaluate next.
What feelings do subjects report for the specific topics? In particular, which feelings are salient or discriminative of certain topics? To analyze this, we look at the prior probability that a feeling gets mentioned across all topics, and the probability that a feeling gets mentioned conditioned on a specific topic. We look at the difference as a measure of how salient a feeling is for a particular topic. See Fig.6. We see that being happy is reported more frequently (6% absolute) in the context of family than prior, but less frequently (10% absolute) than prior in the context of work. Being satisfied and proud is reported more often than prior for exercise, calm for god, frustrated, anxious, sad and bored for work. Being proud is mentioned less often in the context of food than prior, and excited less often for family than prior.
To get a glimpse into the free-form text subjects provided, we visualize word clouds of the entries broken down by topic and feelings. See Fig. 7 for the topics word clouds. Family mentions “time”, “together”, “dinner” and “home”, food mentions “delicious”, “favorite”, “restaurant”, work talks about “project”, “job”, “satisfied”, “money”, exercise about “workout”, “running”, “routine”, “goals”, sleep about “bed“, “night”, “enough”, “refreshed”, and so on. See Fig. 8 for word clouds for each feeling. When subjects reported feeling nostalgic they talked about “dinner” and “childhood”, and about “work” when they reported feeling proud.
We generate a lemotif for each of these 100 journal entries using the approach described earlier. We now describe our experimental setup for evaluating these lemotifs.
We first evaluate our topic-icon and feeling-color mappings.
We showed subjects on AMT the list of 11 topics and (a randomly ordered list of) the 11 icons shown in Fig. 2. Subjects were asked to assign each icon to exactly one topic, ensuring every topic has been assigned an icon. In other words, subjects were asked to perfectly match the 11 icons to 11 topics. We had 170 subjects perform this task. Given a topic, we find that the right icon was picked for it 69% of the times (mean across subjects). Note that chance performance would be 9%. If we assign a topic to the icon that was picked most often for that topic (majority vote across subjects), the accuracy is 82%. For a given topic, we sort all icons by how often they were selected across subjects. We find that the right icon falls at rank 1.27 out of 11 (on average across topics). The right icon falls in the top 20% of the sorted list 91% of the time across topics, and in the top third of the list 100% of the time. Overall, this shows that subjects find our topic-icon mapping to be quite intuitive and natural.
We run a similar study to evaluate our feeling-color mapping shown in Fig. 3. This is a more challenging task because (1) icons have descriptive shapes that can be recognized as objects with semantic meaning, while colors are significantly more ambiguous, and (2) there are 18 feelings and colors as opposed to fewer topics and icons. Note that the choice of colors (and icon) being intuitive and natural to users is a bonus, but not a requirement; as seen in Fig. 1, the topics and feelings are explicitly listed on the lemotif. We run our study on 99 subjects. We find that given a feeling, the right color was picked 15% of the times (mean across subjects). Chance performance would be 6%. If we assign a feeling to the color that was picked most often for that feeling (majority vote across subjects), the accuracy is 33%. For a given feeling, we sort all colors by how often they were selected across subjects. We find that the right color falls at rank 5.28 out of 18 (on average across feelings). The right color falls in the top 20% of the sorted list 61% of the time across feelings, and in the top third of the list 67% of the time. Overall, this shows that in spite of the mapping being this ambiguous and subjective, subjects do find an intuitive and natural signal in our feelings-color mappings to quite an extent. As mentioned earlier, users can be allowed to specify their own feeling-color and topic-icon mappings.
We now discuss our main evaluation. Recall that the underlying principles in Lemotif are that the lemotif should (1) separate out the sources of the emotions, (2) depict these sources visually, (3) depict the emotions visually, and (4) have a creative aspect to them. In this subsection, we validate this hypothesis. To this end, we design several baselines that allow us to measure the role of each of factors. Our proposed lemotif (either circle packing Fig. 4 A1 or string doll Fig. 4 A2) has all four factors. We strip away one factor at a time to derive our various baselines:
We start with our lemotif and remove the shape depiction (still keeping the creative aspect, color depictions, and topic breakdown). We replace each icon in the lemotif with a square. The square can be depicted either via circle packing or the string doll rendering. This gives us two baselines (B1 and B2) in Fig. 4.
We can also start with our lemotif and remove the creative aspect, while maintaining the shape and color depictions, as well as the topic breakdown. We color each shape with solid colors associated with the feelings mentioned for that topic. This gives us B3 in Fig. 4.
We can now remove the shape information from the above baseline, and depict squares (instead of icons) for each topic colored in with solid colors (no creative aspect). This gives us B4 in Fig. 4.
We can start with B3 and remove the detailed color information. Instead of using a color for each of the 18 feelings, we use just three colors: red, yellow, and green to depict negative, neutral or positive feelings. We mapped afraid, angry, anxious, ashamed, disgusted, frustrated, jealous and sad to negative; awkward, bored, calm, confused, nostalgic and surprised to neutral; and excited, happy, proud and satisfied to positive. We use the majority label across reported feelings to pick a color for that topic. This gives us B5 in Fig. 4.
We can remove shape information from the above baseline to have squares colored in either red, yellow or green representing each topic. This gives us B6 in Fig. 4.
Finally, we can remove the topic breakdown from the above baseline and have the entire day depicted as a red, yellow, or green square based on the most common label across reported feelings for the day. This gives us B7 in Fig. 4. As mentioned in the related work section, this mimics an existing app (Life Calendar) that shows a single colored dot for every week in the year.
These 7 baselines and our 2 proposed lemotifs (Fig. 4 A1 and A2) give us 9 approaches to compare. We generate these 9 visualizations for all 100 journal entires in our dataset. We conduct pairwise evaluations on AMT. We show subjects a journal entry from our dataset, and all pairs of visualizations. For each pair, we ask them “If you were using a journaling tool or app that automagically produced a visual summary of your day, which one of these visualizations would you prefer?”. We had 840 unique subjects participate in this study, each providing us a rating for the 36 pairs for a single journal entry. Each journal entry was evaluated by 6 to 10 subjects, with an average of 8.4 and mode of 9.
By comparing appropriate pairs of the 9 approaches we can evaluate the role of each of the 4 factors listed above. Checking how often subjects pick B6 over B7 tells us how important it is for the lemotif to have a breakdown across topics (topic). Similarly, comparing B5 to B6, B3 to B4, A1 to B1, and A2 to B2, we can evaluate the role of a topic being depicted by a shape as opposed to a generic square (shape). Comparing B3 to B5, and B4 to B6, we can evaluate the role of each feeling being depicted by a nuanced color as opposed to a coarse color for negative, neutral, and positive feelings (color). Comparing A1 to A2 tells us which of the two creative lemotifs subjects prefer. We find that subjects prefer circle packing (A1) to string doll (A2) 72% of the time.888A couple of quotes from subjects who did not like the string doll rendering “I don’t like the squigly ones at all!”, “ I really didn’t like the Jackson “Polloky" yarn things. They looked more like knots and were sort of aggitating rather than happy and excited.” A comment from someone who did prefer string doll “The dots and different sizes of dots makes me a little confused about drawing out the different emotions in the day. I like the viz in [string doll] because it gives me a sense of positive and neutral things “blended” in the day.” Most comments from subjects were about these visualizations being “delightful”, “refreshing”, “fun”, “artistic”, etc. We focus our evaluation of the creative aspect on A1. Comparing A1 to B3, and B1 to B4 we can evaluate how much subjects prefer the creative aspect (creative).
In Fig. 9, for each of the four factors, we show how often a visualization with that factor is preferred over a corresponding visualization without that factor (as described above). We show these statistics separately for subjects who were consistent in their preferences vs. those that had some contradictory preferences. Recall that we had each subject report their preferences for all visualization pairs for a single journal entry. We can check whether the pairwise preferences reported are consistent across the board or not (if a b and b c, then a should be c). Presumably, subjects who provide consistent preferences are likely to be doing the task more carefully and/or have more clear preferences. We find that 34% of our subjects were perfectly consistent across the 36 pairwise comparisons. Across the board in Fig. 9
, the the four factors are preferred. This is further exaggerated for subjects who were consistent in their responses. The dashed line at 50% is no preferences (random selections). Error bars are 95% confidence intervals999
It is also clear that there is a variance in preferences. In practice, an app would allow users to pick a visualization they prefer..
We had additionally asked each subject whether they are interested in journaling or not, whether they regularly journal or not, and if yes, whether they use an electronic journal or a physical one. We analyzed the above trends across these different groups of subjects. We find that all four factors are favored more by subjects not interested in journaling. The difference is not statistically significant except in the case of shape. There can be two interpretations of this: subjects interested in journaling, who are perhaps the more natural “customers” for Lemotif, are less keen on these four factors (note that even they do strongly prefer having these factors than not, just a little less than those not interested in journaling). The optimistic view is that perhaps Lemotif can be a good tool to encourage those currently not interested in journaling to begin! Among those interested in journaling, we find no difference in preferences between those who do and do not journal regularly along the topic and color factors. Those who do not journal prefer shape slightly more than those who do (not statistically significant). However, those who do journal prefer creative statistically significantly more than those who do not journal regularly. Again, those who do not journal do prefer creative over not, but those who journal regularly prefer it even more. Finally, among those who journal regularly, we find that those who use an electronic journal prefer creative and shape statistically significantly more than those who use a physical journal. This is promising because Lemotif is (clearly) a computational / electronic tool. Note that the only approaches that we tested that had creative and shape (A1, A2), also had color and topic. That may have contributed to not seeing an affect for color and topic across these groups. Evaluating a circle packing or string doll visualization but in one of the three colors (red, yellow, green) could help disentangle this.
One caveat that we would like to point out is that the above evaluation involves a subject evaluating a lemotif for someone else’s journal entry. Running an evaluation where a subject on AMT evaluates a lemotif from their own journal entry would require an in-browser real-time Lemotif system which is part of future work.
As an alternative, we had 6 subjects in-house (friends, family, colleagues) send us their journal entries. We sent them the 9 visualizations and asked for a sorting according to their preferences. On average, A1 was the most preferred. Among 6 relevant pairwise comparisons, split by topic was preferred all 6 times. Among 24 comparisons, topics being depicted by shape was preferred 20 times. Feelings being depicted by color was preferred 4 out of 12 times (more subjects would be required to infer a meaningful “majority” preference) and a creative element was preferred 10 out of 12 times. 1 subject preferred A2 over A1.
The real evaluation of a system like Lemotif is to see if users journal more regularly or continue journaling for longer, if they feel more creative, if they engage with friends and family more by sharing their lemotifs, and really, if they can decipher actionable patterns through these lemotifs (analyzed over weeks or months) to make changes to their life and be happier. Such an evaluation is outside the scope of this paper. As a proxy, we run a survey on AMT.
We described the idea to subjects and showed them example circle packing lemotifs that covered all topics (icons) and feelings (colors). We asked them a series of 19 questions (including interest in journaling, whether they regularly used an electronic or physical journal). 100 subjects took our survey. 80 were interested in journaling. We share some statistics next (across all subjects, and only for the group interested in journaling). (75%, 78%) thought such an app should exist, (59%, 68%) said they would use it, (26%, 33%) said they would pay $0.99/month for such an app, (4%, 5%) said they would pay $9.99/month (options were $0.99/month, $9.99/month, $19.99/month), (68%, 73%) said Lemotif would make journaling more enjoyable, (78%, 79%) said they would be curious to see what their lemotif for the week or longer durations of time looked like, (65%, 71%) said they would be willing to enter their journal entry in the specific format we currently dictate to be able to generate a lemotif, (61%, 70%) said they think they would be more likely to journal regularly with an app like Lemotif, (40%, 43%) said they would share their lemotif with friends and family, (18%, 19%) said they would share it on social media, (59%, 64%) said Lemotif made their creative juices flow, only (22%, 22%) said they would like to add more topics to the list of 11, (14%, 16%)101010The statistic reported earlier that 99% and 100% subjects reported the 11 topics and 18 feelings being sufficient was based on subjects writing their journal entries. This survey is among subjects looking at lemotifs. The former statistic may be more reliable because it is grounded in what subjects actually want to talk about. said they would like to add more feelings to the list of 18 demonstrating that our list of topics and feelings is fairly comprehensive, (38%, 36%) said they would like to be able to chose their own shapes to represent topics, (49%, 49%) said they would like to be able to choose their own colors to represent feelings, and finally, (33%, 34%) said they would like to be able to choose their own rendering design. In the in-house study, 5 of 6 subjects said they would use such an app and that it would make journaling more enjoyable. 1 subject said they would be willing to pay $0.99/month for the app.
In summary, we presented Lemotif. It takes in as input a journal entry of a user indicating what aspects of the day were salient and how they made them feel, and generates as output a motif – a creative abstract visual depiction – of the user’s day. lemotifs generated for periods of time can make associations between feelings and parts of a user’s life apparent to the user, presenting opportunities to take actions towards improved emotional well being.
Lemotif was built on four underlying principles: A lemotif should (1) separate out the sources of the emotions, (2) depict these sources visually, (3) depict the emotions visually, and (4) have a creative aspect to it. We verified via human studies that each of these factors contributes to the proposed lemotifs being favored over corresponding baselines. We also found that subjects are interested in using an app like Lemotif (and in some cases, even paying for it!).
Thanks to Ayush Shrivastava, Gauri Shri Anantha, Abhishek Das, Amip Shah, Sanyam Agarwal, Eakta Jain, and Geeta Shroff for participating in the study. Special thanks to Abhishek Das for useful discussions and feedback. At Facebook AI Research, we understand that researching a topic like emotion is nuanced and complicated. This work does not research what causes emotional well being (or not). It does not mine Facebook data to extract emotions, or use Facebook data or the Facebook platform in any other way. It simply generates visualizations based on topics and emotions reported by subjects explicitly electing to participate in our study, and analyzes which visualizations subjects prefer. Creative applications of AI are a powerful avenue by which AI can collaborate with humans for positive experiences. This work is one (small) step in that direction.
A Creative Artificial Intelligence System to Investigate User Experience, Affect, Emotion and Creativity.In Proceedings of the Conference on Electronic Visualisation and the Arts.