Jurassic Mark: Inattentional Blindness for a Datasaurus Reveals that Visualizations are Explored, not Seen

08/09/2021
by   Tal Boger, et al.
UNSW
Yale University
Northwestern University
0

Graphs effectively communicate data because they capitalize on the visual system's ability to rapidly extract patterns. Yet, this pattern extraction does not occur in a single glance. Instead, research on visual attention suggests that the visual system iteratively applies a sequence of filtering operations on an image, extracting patterns from subsets of visual information over time, while selectively inhibiting other information at each of these moments. To demonstrate that this powerful series of filtering operations also occurs during the perception of visualized data, we designed a task where participants made judgments from one class of marks on a scatterplot, presumably incentivizing them to relatively ignore other classes of marks. Participants consistently missed a conspicuous dinosaur in the ignored collection of marks (93 where the incentive to ignore that collection was removed (25 presentation, and 11 "seen" in a single glance, and instead require an active process of exploration.

READ FULL TEXT VIEW PDF
12/23/2021

Explaining with Examples: Lessons Learned from Crowdsourced Introductory Description of Information Visualizations

Data visualizations have been increasingly used in oral presentations to...
07/25/2019

Searching the Visual Style and Structure of D3 Visualizations

We present a search engine for D3 visualizations that allows queries bas...
01/15/2019

Spatial Filtering Pipeline Evaluation of Cortically Coupled Computer Vision System for Rapid Serial Visual Presentation

Rapid Serial Visual Presentation (RSVP) is a paradigm that supports the ...
11/04/2021

Defining Gaze Patterns for Process Model Literacy – Exploring Visual Routines in Process Models with Diverse Mappings

Process models depict crucial artifacts for organizations regarding docu...
03/14/2021

Active Dynamical Prospection: Modeling Mental Simulation as Particle Filtering for Sensorimotor Control during Pathfinding

What do humans do when confronted with a common challenge: we know where...
12/07/2021

VizExtract: Automatic Relation Extraction from Data Visualizations

Visual graphics, such as plots, charts, and figures, are widely used to ...

1 Related work

Now a staple of the research literature on attention and perception, inattentional blindness was greeted with some astonishment even from seasoned experts in the field when it began to be widely reported in the 1990s [mack1998inattentional, simons1999gorillas]. Earlier versions of the phenomenon, referred to as “selective looking”, date back to the 1970s; [becklen1983selective, neisser1975selective, neisser2019control].) In a classic set of studies, people reported whether the horizontal or vertical portion of a cross was longer on each trial and failed to notice even highly conspicuous - but unexpected - items that appeared in one of the cross’s quadrants on a critical trial [mack1998inattentional]. The astonishment over such findings stemmed not only from the fact that they seemed to run counter to lay intuition that seeing is a function merely of the eyes, but also from the fact that much evidence until then had suggested that some salient features should be noticed without the need for attention [treisman1988feature].

Figure 1: Types of plots shown in our tasks, along with an example dinosaur.

Work on inattentional blindness has shown both of these assumptions to be incorrect. For example, using eye-trackers, researchers found that people often fail to see the unexpected objects despite looking directly at them [beanland2010looking, koivisto2004effects, memmert2006effects]. Building on a particularly well-known demonstration of inattentional blindness, where participants failed to see a person in a full-body gorilla outfit walk through the middle of a videotaped group of people passing basketballs [simons1999gorillas], one study found that people who missed the gorilla looked at it for about the same amount of time as did the people who saw the gorilla [memmert2006effects].

Meanwhile, notions that some visual features should be seen without attention had grown out of research on “visual search”, in which participants sought targets that were embedded in arrays containing varying numbers (or “set sizes”) of non-target elements. Although it typically takes participants longer to find the target when it is embedded within a large array than within a small array, targets that were defined by a particularly unique or salient feature tended to be found quickly regardless of the size of the array, suggesting that they bypassed the need for attention, a phenomenon known as “visual pop-out” [treisman1988feature, wolfe1994guided]. Thus, subsequent findings that 28% of participants were inattentionally blind to a red shape that was fully visible for 5 seconds among black and white shapes (one set of which participants were tracking; [most2001not]) and that 32% were inattentionally blind to a white shape visible for 5 seconds among uniformly black shapes [most2005you] contrasted with assumptions that such unique and salient features should push their way into participants’ awareness [mack1998inattentional].

Why would features that appear to bypass the need for attention in visual search tasks fail to be noticed under conditions of inattentional blindness? A key difference between these tasks is that people know to expect the critical item in visual search tasks, whereas the critical item is unexpected in experiments on inattentional blindness. Thus, in visual search tasks, attention is broadly directed to the visual display in search for the target. Indeed, work on inattentional blindness has repeatedly underscored the importance of how people tune or direct their attention: when the unexpected item contains features similar to those that people are actively attending to, people notice it much more than when its features are similar to the items that people try to filter out. For example, when people tracked moving black items and ignored moving white items, they were far more likely to notice an unexpected and unique black than white shape that traversed the display, with the reverse pattern emerging among those who tracked the white items; in both cases, noticing of a gray shape fell in between [most2001not, most2005you, simons1999gorillas]. In short, much of what we see depends on how our attention filters the world around us.

Research on visual attention has clear connections to visualization perception, which often requires attending to some trends and ignoring others. For example, change blindness – a failure to detect salient changes to a scene due to capacity limits and comparison mechanisms – presents powerful constraints for displays where viewers make visual comparisons between separate charts [gleicher2011visual, nowell2001change].

2 Experiment 1: filtration task

In Experiment 1, participants completed a task that required them to focus on a signal class of marks (blue X’s) in a scatterplot, while selectively filtering out blue and green circles. Based on existing demonstrations in the perceptual psychology literature, we hypothesized that this filtering requirement would cause sufficient inhibition of the other marks that participants would fail to notice the dinosaur shape.

2.1 Stimuli

All stimuli were created in R using ggplot2 [ggplot2]. The scatterplots contained three classes of marks, as seen in Figure 1. The dinosaurs were plotted as green circles. To maximize the incentive for viewers to selectively inhibit both the green color and circle shapes, we asked them to attend only to blue X’s (to encourage inhibition of green and circles), and added a set of to-be-ignored blue circles (to additionally encourage inhibition of circle shapes).

We adapted the coordinates of the dinosaur shape from an image designed by Alberto Cairo [cairo1970]

, and centered their coordinates to have mean 0 and standard deviation 1 within our arbitrary coordinate space. Because people can detect the statistical properties of collections of objects even when those collections are not selectively attended (e.g., mean position; see

[alvarez2008representation]

), we maximized the statistical similarity among the three sets of marks on the display. So, each group — including the green circles in plots where there was no dinosaur — was drawn from a normal distribution with the same mean, standard deviation, and number of samples in each quadrant as the dinosaur had.

2.2 Task and procedure

On each trial in this experiment, participants saw a series of seven graphs in quick succession. In Experiment 1a, each plot appeared on screen for 1s, and then disappeared for 250ms before the next one appeared. In Experiment 1b, we extended this time, such that plots stayed on screen for 2.5s, and then disappeared for 250ms.

In each series of seven plots, participants were asked to determine whether there were more plots in which the blue X’s created a forward-slash (i.e., the blue X’s were only in quadrants 1 and 3; plot type 1 in Figure 1) or more plots in which the blue X’s created a backward-slash (i.e., the blue X’s were only in quadrants 2 and 4; plot type 2 in Figure 1). To ensure participants could not “cheat” by determining whether a plot created a forward-slash or back-slash by looking only at only the top half (or any other half) of the graphs, we created a third plot type, in which the blue X’s were either in only one quadrant or in three quadrants (plot type 3 in Figure 1).

Each series of seven plots contained either three plots with forward-slashes and two plots with back-slashes, or three plots with back-slashes and two plots with forward-slashes (chosen randomly for each participant). The remaining two plots were distractors (plot type 3); in one of the two, the blue X’s were present only in one quadrant, and in the second the blue X’s were present in three quadrants. The order of the plots in each sequence was randomized for each trial.

Participants completed five trials in which they saw the sequence of plots described above. On both the third and fifth trials, one of the plots in the sequence contained the dinosaur shape within the green circle marks. The dinosaur was randomly chosen to appear in any plot type (a forward-slash plot, back-slash plot, or neither). The position of the dinosaur within the third and fifth trial sequence was randomly chosen between the fourth and sixth plot of each of the sequences.

After completing the experiment, participants were asked whether they noticed any patterns in any of the groups of data points. Then, they were asked explicitly whether they noticed a dinosaur in any of the plots, producing our dependent variable. Finally, they were also asked whether they noticed a gorilla in any of the plots. However, no plots contained a gorilla — we added this question to account for the potential that participants would claim to notice things without actually seeing them.

Readers may try all tasks presented in this paper for themselves at https://datasaurus-vis.herokuapp.com/

2.3 Participants

For each of Experiment 1a and Experiment 1b, we recruited 20 participants from the online recruiting platform Prolific. For a discussion of the reliability of this subject pool, see [peer2017beyond]. Participants were compensated upon completion of the experiment at a rate of $9/hour.

2.4 Results

We excluded 1 participant in Experiment 1a who did not submit a complete dataset. We also excluded 4 participants in Experiment 1a, and 2 participants in Experiment 1b, whose accuracy in the given task was below chance (50%), under the assumption that these participants were not paying attention to the task, or simply could not perform it.

These exclusion criteria left us with 15 participants in Experiment 1a, and 18 participants in Experiment 1b. In Experiment 1a, when the plots were visible for 1000ms, 14/15 (93.3%) of participants missed the dinosaur. Furthermore, the single participant who claimed to see the dinosaur also claimed to see the gorilla as well as other patterns, none of which were actually present. In Experiment 1b, when the plots were visible for a full 2.5s, 11/18 (61.1%) of participants missed the dinosaur. Two of the participants who claimed to see the dinosaur also claimed to see other things (e.g. the gorilla) that were never present.

In both experiments, participants missed the dinosaur at high rates, suggesting that the filtering requirements of the scatterplot judgment task led to inattentional blindness in the data visualization.

3 Experiment 2: control

The results of Experiments 1a and 1b suggest that the filtering requirements of the task caused participants to miss the dinosaur. Yet, it is also possible that the dinosaur was simply too hard to see, perhaps overcrowded by other points in the display, or appeared too quickly to be perceived. Intuitively, viewing a trial suggests that neither is true, as the dinosaur is easily visible, and is present for a full 2.5s in Experiment 1b. Therefore, Experiments 2a and 2b test a control condition that relies on identical displays and timing, but selectively removes the filtering requirement of the task. If the filtering per se were responsible for the missed dinosaur, then miss rates should fall substantially in the control conditions.

3.1 Stimuli

We used the same stimuli as in Experiments 1a and 1b.

3.2 Task and procedure

The plot sequence, randomization, and parameters here were exactly the same as they were in Experiments 1a and 1b. Participants again saw five trials in which a sequence of seven plots appeared, with the same randomization and plot types as before. In Experiment 2a, the plots appeared for 1000ms (as in Experiment 1a), and in Experiment 2b, they appeared for 2500ms (as in Experiment 1b). The dinosaur again appeared on the third and fifth trial sequences, with the same randomization as in 1a and 1b. The key difference was that participants were not told to count forward-slashes and back-slashes within the blue X’s. Instead, they were told simply to watch the plots for “anything unusual.” As before, at the end of the experiment, participants were asked whether they noticed any trends, and whether they saw a dinosaur or gorilla.

3.3 Participants

For each of Experiment 2a and Experiment 2b, we recruited 20 new participants from Prolific.

3.4 Results

Because there was no participant task that could produce objectively correct answers, we did not exclude participants in these experiments based on accuracy, as we did before. We excluded 1 participant in Experiment 2b who did not submit a full dataset due to a technical error.

In Experiment 2a, when the plots were visible for 1s, 5/20 (25%) of participants missed the dinosaur (one participant who claimed to see the dinosaur also claimed to see the gorilla), compared to Experiment 1a where 93.3% of participants missed the dinosaur. In Experiment 2b, when the plots were visible for 2.5s, only 2/19 (10.5%) of participants missed the dinosaur (five participants who claimed to see the dinosaur and also claimed to see the gorilla), compared to 61.1% in Experiment 1b. There was a significant relationship between the rate of missing the dinosaur and the task type (i.e., filtration vs. control) both when comparing Experiment 1a to Experiment 2a () and when comparing Experiment 1b to Experiment 2b ().

Figure 2: Rates of inattentional blindness (percent of participants who missed the dinosaur) by task and plot speed.
Figure 3: A plot showing how designers can effectively communicate data to limit filtering operations. Some designers might default to creating only the two left-most plots. However, including the third plot – which shows differences between groups – reduces the number of comparisons a reader needs to make and points them to the “right” pattern in the data.

4 Discussion & future work

In two experiments, we found that when inspecting a pattern in a single class of marks (blue X’s), the resulting visual filtering operation was so powerful that participants missed an otherwise clearly visible dinosaur shape made of relatively inhibited marks (green circles).

4.1 Implications for designing data visualizations

The failure to perceive a salient and surprising pattern among a relatively ignored set of marks has practical implications for designing effective and informative data visualizations. First, visualization designers cannot assume that the same patterns that are salient to them will be salient for other viewers. In fact, recent work shows that viewers sometimes assume that others will focus on the same patterns that they do in a visualization, while in reality, each person can have their own idiosyncratic set of filters for what patterns they focus on [xiong2019curse]. Designers might consider simplifying the dimensionality of the data that they show, reducing the space of possible filtering operations (and resulting comparisons) that a viewer needs to make, or at minimum, using annotation and highlighting to help a viewer see critical patterns instead of assuming that they will be noticed [ajani2021declutter].

For example, consider a designer who wants to show how two treatments affect infection rates differently across age groups. Specifically, the designer wants to show that treatment A seems more effective than B in preventing infection for younger and older patients, but that B is more effective for patients 30-50 years old. One could show the data as two juxtaposed bar graph series, using only the left and center columns of Figure 3. However, that design requires the reader to make second-order comparisons of the (A-B) delta values for each age group. This requires multiple filtering operations that might be obvious to the visualization’s designer, but not for the novice viewer [xiongVisual]. A better design might facilitate extraction of the critical pattern by including a third series that depicts the deltas directly, as in the right-most plot of Figure 3 [nothelfer2019measures].

4.2 Directions for future work

In the experiments presented here, the trend that participants missed was purely visual (the dinosaur formed by the green points). This presents a case where the trend missed is visually obvious and unique at a perceptual level, and should be remembered if it had been noticed. Another strategy for future work might be to seek similar cases where the surprising pattern is not perceptual, but conceptual, by showing that viewers can miss a relatively simple pattern that otherwise clearly invalidates an argument, or the validity of the dataset, in a way that should be easy to spot if the pattern were noticed (e.g., data revealing that a child appears to be older than their parent). Such a demonstration would help demonstrate the power of progressive visual filtering operations with a dependent measure more tied to the types of sensemaking and decisions that visualizations are meant to produce [lee2015people].

Future work may also explore inattentional blindness in data visualizations parametrically in ways beyond those presented here. Between our experiments, the only parameter that changed was the amount of time the plots stayed on the screen (1000ms in Experiments 1a and 2a, compared to 2500ms in Experiments 1b and 2b). However, one could also manipulate the number of distractor points, the number of total groups (i.e., adding or removing a distractor group entirely), or various visual features of the existing points (e.g., making the points for the dinosaur group thicker). Exploring this parameter space might shed light on the power and limits of visual filtering operations in realistic data displays.

5 Conclusion

Our experiments revealed consistent inattentional blindness for data visualization, lending strong support to the idea that data visualizations are explored over time rather than instantly seen. When tasked with performing a series of filtration operations over time, participants missed an obvious dinosaur at a significantly higher rate than when they were told to simply look at the plots for anything unusual. These results strengthen the evidence for the design guideline that visualization viewers should be guided toward the “right” pattern in a communication context. Otherwise, if they fail to filter for the right type of information, even highly salient and important patterns can be missed.

References