An Experimental Comparison of Map-like Visualisations and Treemaps

by   Patrick Cheong-Iao Pang, et al.
University of Macau

Treemaps have been used in information visualisation for over two decades. They make use of nested filled areas to represent information hierarchies such as file systems, library catalogues, etc. Recent years have witnessed the emergence of visualisations that resemble geographic maps. In this paper we present a study that compares the performance of one such map-like visualisation with the original two forms of the treemap, namely nested and non-nested treemaps. Our study employed a mixed-method evaluation of accuracy, speed and usability (such as the ease-of-use and helpfulness of understanding the information). We found that accuracy was highest for the map-like visualisations, followed by nested treemaps and lastly non-nested treemaps. Task performance was fastest for nested treemaps, followed by non-nested treemaps, and then map-like visualisations. For usability, nested treemaps was considered slightly more helpful than map-like visualisations while non-nested performed poorly. We conclude that the results regarding accuracy are promising for the use of map-like visualisations in tasks involving the visualisation of hierarchical information, while non-nested treemap are favoured in tasks requiring speed.


page 3

page 4

page 5

page 8


On a conjecture by Ben-Akiva and Lerman about the nested logit model

We prove a conjecture of Ben-Akiva and Lerman (1985) regarding the rando...

LibNDT: Towards a Formal Library on Spreadable Properties over Linked Nested Datatypes

Nested datatypes have been widely studied in the past 25 years, both the...

Hierarchical Reinforcement Learning with Deep Nested Agents

Deep hierarchical reinforcement learning has gained a lot of attention i...

Model Selection of Nested and Non-Nested Item Response Models using Vuong Tests

In this paper, we apply Vuong's (1989) general approach of model selecti...

Nested Aggregates in Answer Sets: An Application to a Priori Optimization

We allow representing and reasoning in the presence of nested multiple a...

Randomized Nested Polar Subcode Constructions for Privacy, Secrecy, and Storage

We consider polar subcodes (PSCs), which are polar codes (PCs) with dyna...

A Neural Transition-based Model for Nested Mention Recognition

It is common that entity mentions can contain other mentions recursively...

1 Introduction

Information visualisation is a powerful tool for people to understand information hierarchies such as file systems and library catalogues, particularly those in which information is buried deep within lower levels of the hierarchy. It also provides means for a user to leverage the power of human perception to analyse and reason about the data Hall2013. Not only can visualisation show structure and relationships inherent in the information, but also summarise and transform it into a flattened representation that reduces information overload Wills2009. Among the many forms of information visualisation, treemaps have been widely applied to information hierarchies. Relatively more recent is the development and application of map-like visualisations to hierarchical information. These represent information structures in a form resembling a geographic map. This paper compares the performance of map-like visualisations with treemaps in visualising information hierarchies.

The treemap JS:1991; Shn:1992 and its variants Bruls2000; Wijk1999; Schreck2006; Liang2012 have for many years been popular in displaying hierarchical and relational data and are found in many application domains Tu2007.

Map-like visualisations BAY2015; YBA:2015; Pang:2011, which have recently received more attention, display hierarchical data and resemble geographic maps. In this category of visualisation hierarchical entities are mapped to visual map elements such as countries, provinces and counties. In this way, the overview and the organisational relationships of the underlying data is visible in a map metaphor, which requires no prior training for it to be understood BAPS2014; PBY2016. In addition, in a recent study map-like visualisations were found to be easy to read and enjoyable to use BAY2015.

Despite the documented advantages of map-like visualisations in helping users perceive and use the hierarchical data they represent BAY2015; PBY2016, the performance and the usability of such visualisations remains unclear. To the best of our knowledge, there is no previous research comparing map-like visualisations with other forms of visualisation. Therefore we set out to compare map-like visualisations with treemaps, one of the most well-established and widely-used forms of visualisation for hierarchical data. Our comparison focused on accuracy and speed of task performance. Extending from our previous work BPP2017, we further report on an enhanced statistical analysis, a test on usability, and a qualitative analysis on the user feedback. Among the various treemap algorithms we selected the original two types of treemap proposed by Johnson and Shneiderman in the first published treemap paper, namely the nested and non-nested treemap JS:1991.

For this research we recruited 40 participants to complete an evaluation involving the use of three types of visualisation (non-nested and nested treemap, and map-like) and to identify the relationship between entities depicted in the visualisation. Our results showed that the accuracy was highest for map-like visualisations, followed by nested treemaps and non-nested treemaps, and that these findings were statistically significant. However, both types of treemaps were better at supporting faster task completion compared with map-like visualisations. Additionally, for usability, both nested treemaps and map-like visualisations are competent in helping readers to understand the underlying data. After analysing the qualitative feedback from our participants, we conclude that the results are promising for sing map-like visualisations as an alternative in representing hierarchical information.

2 Related Work

2.1 Treemap Visualisations

Originally invented by Shneiderman and his colleagues in the 1990s JS:1991; Shn:1992, treemap visualisations have been extensively investigated and widely used in many application areas. In order to overcome some of their drawbacks, such as readability, stability and ordering of blocks, a number of variants of treemaps have been proposed, including Squarified Treemaps Bruls2000, Cushion Treemaps Wijk1999 and Ordered and Quantum Treemaps Bederson2002. However, few of these algorithms aim at improving the application of treemaps to hierarchical data.

Figure 1: Example of a Cushion Treemap Kosara (image licensed under CC-BY-SA)
Figure 2: Example of a Squarified Treemap Kosara (image licensed under CC-BY-SA)

An obvious problem of treemaps is that the hierarchical organisation of data, particularly in cases of deep nesting, cannot be depicted clearly in the visualisation Tu2007; Liang2012. Researchers have proposed different methods for improving the readability of nested data in treemaps. For instance, Cushion Treemaps use shadings and colours to represent areas belonging to the same parent (Figure 1); and Squarified Treemaps show hierarchical data in nested square blocks (Figure 2). These approaches allow the use of treemaps with deeply nested data, and have been adopted to some actual visualisation problems Kosara, including the visualisation of disk usage that originally motivated the invention of treemaps. Other approaches for visualising large trees include the use of non-rectangular areas, such as in Divide and Conquer Treemaps LNSH2015.

Stability and comparability are drawbacks of some treemap algorithms that are used to visualise hierarchical data. Some small differences in the hierarchy can cause significant changes to the layout of treemaps Hahn2015, which makes it very hard to compare one version of the data against another. Some extensions of the treemap visualisation have been proposed to address this problem. One way is to limit the aspect ratio of the regions in treemaps deBerg2014, as extreme aspect ratios are difficult to compare. Another way is to extend the treemap algorithm by using the properties of Voronoi diagrams, so that changes are only reflected locally in a smaller area Hahn2014. Such work makes treemaps more useful in applications that require comparison, for example organisation charts and source code repositories.

2.2 Map-like Visualisations

Map-like visualisations display hierarchical data in the form of a geographic or topographic map, and are also known as metaphoric maps CP:2012. Cartographic methodologies are employed in generating these visualisations. A common approach to displaying hierarchical data with map-like visualisations is to depict data as cartographic elements (e.g. land and sea) and related different types of data points to map elements Couclelis1998. In general, multiple levels of nested data are conventionally shown as nested areas in a map (e.g. countries, provinces, counties, districts, etc). As a result lay users can easily perceive the information contained in the visualisations, as studies have confirmed that most average readers can effectively read and understand maps in their daily lives Borner2003; Borner2010.

(a) Skupin’s SOM map, reproduced from Skupin2004
(b) GMap, reproduced from Mashima2012
(c) Topological map-like visualisation, reproduced from Gronemann2013
(d) GosperMap, reproduced from Auber2013
(e) Hexagon tiling-based map-like visualisation, reproduced from BAPS2014
(f) Liquid modelling-based map-like visualisation, reproduced from BAY2015
(g) A visualisation with probabilistic graph layout, reproduced from Schulz2017
Figure 3: Examples of map-like visualisations

A number of algorithms have been proposed for creating map-like visualisations for different types of data. Skupin suggested using the self-organising map (SOM) to train a dataset and to visualise the resulting clusters as land masses (Figure 2(a)) Skupin2004. GMap is another algorithm that draws undirected graphs as geographic maps (Figure 2(b)) Gansner2010; Mashima2012

. Gronemann and Jünger have created an algorithm to transform networked graphs to topological map-like visualisations (Figure 

2(c)) Gronemann2013. Auber et al. make use of Gosper curves for laying out data regions in the visualisation image (Figure 2(d)) Auber2013. Recent studies also highlight the use of hexagons for map-like visualisations by tiling coloured hexagons on a surface (Figure 2(e)) BAPS2014; PBY2016. Biuk-Aghai et al. have applied a liquid modelling approach to generating map-like visualisations by emulating the interactions of liquid collisions (Figure 2(f)) BAY2015. Although it is not a kind of map-like visualisations, a recent attempt at visualising data using the probabilistic graph layout (Figure 2(g)) produces output that somewhat resembles a map Schulz2017.

Thanks to the early exposure of maps at an early stage of education Blades1998, they are readily usable by lay users without prior training. As a result, applications of map-like visualisations are growing in various domains. For example, they are being used to illustrate the content and sizes of document corpora BAPS2014 and software packages Auber2013. Additionally, maps allow readers to navigate and explore unknown data Borner2010, which is a recent focus in information visualisation. As a result, visualisations are created to support various information-seeking tasks, such as browsing, orienteering, exploring and interactive query refinement Kairam2015; Khazaei2017. Some map-like visualisation applications have the potential for people to discover educational material Pang:2011; PVPC2015, explore knowledge domains Skupin2004 and analyse data in the medical context Skupin2013; PHMV2016; PCC2017. Given this momentum of extending map-like visualisations to more application areas, it becomes desirable to understand their effectiveness and how they compare with existing visualisation alternatives. The experimental study presented in this paper is one step in this direction.

3 Research Design

Our research aims to compare a map-like visualisation with the well-established treemap visualisation, in terms of their ability to represent hierarchies in an easy-to-understand manner. Specifically, we are interested to assess our assumption that map-like visualisations are more intuitive than treemaps when used by lay users to perform tasks requiring an understanding of hierarchies, and that this will lead to better performance of a map-like visualisation as compared to treemaps, both in terms of accuracy and speed. Thus we set out to test this hypothesis for accuracy (A), presented together with its null-hypothesis:

Hypothesis :

Map-like visualisations allow lay users to perceive hierarchies more accurately than treemaps do.

Hypothesis :

Map-like visualisations do not allow lay users to perceive hierarchies any more accurately than treemaps do.

Likewise, here is the hypothesis for speed (S), together with its null hypothesis:

Hypothesis :

Map-like visualisations allow lay users to perceive hierarchies faster than treemaps do.

Hypothesis :

Map-like visualisations do not allow lay users to perceive hierarchies any faster than treemaps do.

To test these hypotheses we designed a task that involved hierarchical data which we visualised using both a map-like visualisation tool, and two existing open source treemap visualisation implementations, producing a nested and a non-nested treemap, respectively. The hierarchical data set we used was data on student numbers of our university, arranged in a hierarchy from programme (bachelor, master, PhD), through academic unit, department, and major, to year. We collapsed the five levels of this hierarchy into four levels by aggregating the numbers of students by year to total number of students over all years.

3.1 Visualisation Images

We created three groups of visualisation images: one group each of non-nested treemaps, nested treemaps, and map-like visualisations. To avoid the viewer guessing the right answer from the colours or text label within the images we labelled areas with meaningless text, and coloured the background of the entire image in the same colour.

The non-nested treemaps were created using the Protovis software by the Stanford Visualization Group 111 We modified the software to use only one fill colour for all areas in the treemap.

The nested treemaps were created using the JavaScript InfoVis Toolkit by Nicolas Garcia Belmonte222 We used the squarified tiling algorithm and again modified the software to only use one fill colour for all area bodies, and one colour for the area title. This treemap is interactive, so we converted the produced treemap to a static image for use in our evaluation.

Finally, the map-like visualisation images were created using a software developed by us that implements the enhanced hexagon tiling algorithm (EHTA) YBA:2015. We chose this particular algorithm for two reasons: firstly, we have access to the program code, enabling us to generate map-like visualisations of our data; and secondly, in a previous study this particular algorithm was found to produce visualisations that most strongly resemble geographic maps PBY2016. Samples of the images used are shown in Figure 4.

(a) Non-nested treemap
(b) Nested treemap
(c) Map-like visualisation
Figure 4: Samples of visualisation images used in our evaluation

3.2 Survey Design

The survey was conducted in a controlled setting, in a computer lab in our university. The researchers gave a brief introduction about the survey, explaining the concepts of hierarchy and of information visualisation, and introducing the treemap visualisation for representing hierarchies. This was followed by the survey itself which was conducted by each participant at a computer, accessing a survey website prepared by us containing the visualisation images described above. This online survey consisted of five parts:

  1. Informed consent form: explaining that participation is voluntary and requiring participant agreement to continue with the study (as required by our university’s ethics committee).

  2. Entry questionnaire: collecting information on the participant’s age, gender, degree pursued (bachelor, master or PhD), IT skills (on a 7-point Likert scale ranging from “know nothing” to “know how to write computer programs”), and knowledge about information visualisation (on a 7-point Likert scale ranging from “not at all” to “expert”).

  3. Practice: asking the participant to evaluate three visualisation images and to answer questions about the relationship of areas in the image. The evaluation task is explained in more detail below. The purpose of the practice questions was to serve as a warm-up to let participants get used to the questions to be answered in the following main evaluation part of the survey. However, there was no indication in the user interface that this part was a practice only, so participants performed this task as if it counted toward the actual evaluation. Answers collected in this part were not evaluated.

  4. Evaluation: this is the main part of the survey in which the participant answered questions about the visualisation images, in different order by group as defined below. On each page, five images were presented to the participant and for each image the participant was asked to answer one multiple-choice question as explained below.

  5. Exit questionnaire: here we revealed to the participant what the visualisation images represented, namely our university’s student enrolment numbers, and asked for an overall assessment of the visualisation. The overall assessment included the perceived levels of ease-of-use and helpfulness of the visualisations for the given tasks, as well as an open-ended question for collecting qualitative feedback about the visualisations. For qualitative comments, we adopt an open coding approach saldana2015coding to obtain important insights from the data.

To eliminate the effect of ordering on results we divided participants in four groups, each of which evaluated the same sets of images but in different order. The basic ordering was treemap vs. map-like visualisation images, and within the group of treemap images a further ordering was made with nested vs. non-nested images. The order of images evaluated by the four groups of participants were thus as follows:

Group A:

non-nested nested map-like

Group B:

nested non-nested map-like

Group C:

map-like non-nested nested

Group D:

map-like nested non-nested

3.3 Evaluation Task

The evaluation task required participants to recognise the relationship between two areas shown in the visualisation. As areas in the visualisation represent nodes in a hierarchy, this means that the task involved recognising this relationship between the nodes. Figure 5 shows an example illustrating this. Figure 4(a) depicts the structure of a hierarchy, which is a rooted tree. Each node other than the root has a parent node, which is the node immediately above it in the tree, linked to it by an edge; and it may have multiple child nodes, which are nodes below it in the tree, linked to it by edges. We name the parent directly connected to a node its direct parent, and likewise there may be multiple direct child nodes. For example, the direct parent of node I is node G, and the direct children of node I are nodes J and K. The parent of a parent is called an indirect parent, and likewise the indirect parent of a parent recursively all the way up to the root. The child of a child is called an indirect child, and likewise the indirect child of a child recursively down the tree. For example, node M has the direct parent K and the indirect parents I, G and A. Node B has the direct children C and D, and the indirect children E and F.

for tree=circle, draw, minimum size=1.5em, inner sep=1pt [A [B [C [E] [F]] [D]] [G [H] [I [J] [K [L] [M]]]]]

(a) Tree structure
(b) Corresponding treemap
Figure 5: Example hierarchy

The tree structure of Figure 4(a) is represented as the treemap shown in Figure 4(b). In the treemap, parent-child relationships (direct and indirect) are represented through the nesting of areas: parents contain their direct and indirect children, recursively down the hierarchy; conversely, children are contained within their direct and indirect parents, recursively up the hierarchy. For example, in Figure 4(b) area M is contained within area K (its direct parent), which is contained within I, which is contained within G, which is contained within A, these being its indirect parents. Similarly, child relationships are represented in the same way.

A task involving the recognition of the hierarchical relationship within a treemap could for example ask what the relationship between areas M and K is (correct answer: M is a direct child of K); or what the relationship between areas G and K is (correct answer: G is an indirect parent of K); or what the relationship between areas C and D is (correct answer: C and D are at the same level).

Our evaluation task presented an image such as one of those shown in Figure 4 and asked the participant following question:

What is the relationship between A1 and A2?

The possible answers provided were:

  1. A1 is an indirect parent of A2

  2. A1 is a direct parent of A2

  3. A1 is an indirect child of A2

  4. A1 is a direct child of A2

  5. A1 and A2 are at the same level

  6. I don’t know

In the actual evaluation, instead of A1 and A2, the question and answer text showed the names of areas existing in the image, such as MED and PSS in Figure 3(c) (in this case the former is a direct parent of the latter). The terms direct parent, indirect parent, direct child and indirect child had been introduced during the brief introduction prior to the beginning of the survey, so participants were familiar with the meaning of these terms in the context of our evaluation.

3.4 Participant Recruitment

We invited students to join our survey through our university’s student associations, who sent out invitation messages through social media (Facebook and WeChat). These messages reached thousands of students from across all our university’s academic units, majors, and degree programmes. We believe this recruitment process helped ensure that a representative sample of students was recruited. Our invitation message asked students to participate in our research in one of the computer labs on campus, that it would take about 1 hour, and that each student would be rewarded for their participation with a supermarket coupon (of about USD6.25 value). 60 students signed up for our survey, and finally 40 students participated.

4 Results and Discussion

We present results of the demographic survey, of accuracy and speed of task performance. Also, we report on the results of the exit questionnaire regarding to the perceived ease-of-use and helpfulness for understanding the data, as well as the qualitative feedback. In addition, we include a discussion of the results.

4.1 Demographic Results

Degree N %
Bachelor 14 35.0%
Master 21 52.5%
PhD 5 12.5%
Table 1: Degree Pursued by Participants
Academic Unit N %
Arts & Humanities 6 15.0%
Business Administration 6 15.0%
Chinese Medical Science 2 5.0%
Education 1 2.5%
Law 4 10.0%
Social Science 7 17.5%
Science & Technology 14 35.0%
Table 2: Academic Unit of Participants

A total of 40 participants completed the survey, equally divided into Groups A, B, C, and D as explained above, i.e. with 10 participants per group. There were 22 female and 18 male students, with a mean age of 22.6 years and a median age of 23 years. 14 of them were Bachelor students, 21 were Master students and 5 were PhD students (representing 35.0%, 52.5% and 12.5% of participants, respectively). The degree pursued by these students is summarised in Table 1 and our participants came from almost all academic disciplines, with 14 (35%) from science and technology, and 26 (65%) from disciplines such as business, education, law, social sciences and others. their distribution across academic units is shown in Table 2. Participants self-assessed their IT skills ranging from a lowest value of 3 (“know how to use office software and Internet”) to a highest value of 7 (“know how to write computer programs”), with a mean value of 5.2, higher than the mid-point value of 4. In terms of knowledge of visualisation they assessed themselves ranging from a low of 1 (“not at all”) to 6 (between “neutral” and “expert”), with a mean value of 3.6 which is somewhat lower than the mid-point value of 4. Thus this sample of participants was technologically adept, but in terms of knowledge about visualisation could be considered lay users, which is what we assume for our hypotheses.

4.2 Accuracy

We collected responses to the evaluation questions and compared these with the correct answer. Each answer was then mapped to a binary true–false value, i.e. answers were considered either completely right or completely wrong. Tables 3, 4 and 5 show the summary of results for accuracy for non-nested treemaps, nested treemaps, and map-like visualisations, respectively. Table 6 shows the mean values of accuracy for each of the four groups and all three types of visualisation, and Figure 6 shows a plot of these accuracy values. i.e. the values for groups A … D in the last column of Tables 3, 4 and 5.

Group NN1 NN2 NN3 NN4 NN5 NN6 NN7 NN8 NN9 NN10 Avg.
A 10% 10% 20% 10% 20% 20% 40% 40% 0% 40% 21%
B 10% 10% 60% 50% 20% 50% 70% 40% 30% 40% 38%
C 20% 20% 40% 50% 30% 10% 90% 30% 10% 20% 32%
D 10% 10% 10% 40% 10% 20% 60% 60% 10% 50% 28%
Avg. 13% 13% 33% 38% 20% 25% 65% 43% 13% 38% 30%
Table 3: Mean accuracy by group for non-nested treemaps (images NN1 … NN10)
Group N1 N2 N3 N4 N5 N6 N7 N8 N9 N10 Avg.
A 70% 70% 90% 40% 40% 70% 90% 70% 90% 100% 73%
B 70% 30% 80% 40% 60% 80% 90% 100% 90% 100% 74%
C 80% 40% 80% 30% 70% 100% 90% 80% 70% 100% 74%
D 50% 70% 40% 70% 70% 50% 90% 60% 70% 80% 65%
Avg. 68% 53% 73% 45% 60% 75% 90% 78% 80% 95% 72%
Table 4: Mean accuracy by group for nested treemaps (images N1 … N10)
Group M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 Avg.
A 40% 80% 100% 100% 100% 80% 100% 100% 100% 100% 90%
B 20% 90% 90% 80% 100% 30% 100% 80% 80% 80% 75%
C 80% 90% 70% 80% 90% 70% 90% 80% 90% 100% 84%
D 80% 90% 90% 100% 100% 90% 100% 80% 100% 100% 93%
Avg. 55% 88% 88% 90% 98% 68% 98% 85% 93% 95% 86%
Table 5: Mean accuracy by group for map-like visualisation (images M1 … M10)
Group Non-nested Nested Map-like Avg.
A 21% ( 12.9%) 73% ( 10.6%) 90% (5.1%) 61%
B 38% ( 11.6%) 74% ( 13.1%) 75% ( 13.8%) 62%
C 32% ( 10.9%) 74% ( 11.8%) 84% ( 16.1%) 63%
D 28% ( 10.0%) 65% ( 18.3%) 93% ( 7.2%) 62%
Avg. 30% 72% 86% 62%
Table 6:

Mean accuracy by group for non-nested treemaps, nested treemaps and map-like visualisations (ranges of the 95% confidence interval in parentheses)

Figure 6: Mean accuracy by group

The results for accuracy show that for all groups the accuracy of the map-like visualisation was highest, ranging between 75% and 93% mean accuracy per group, followed by the nested treemap (ranging between 65% and 74%), and the non-nested treemap finishing a distant third (21% to 38%). The gap in accuracy between the highest and second highest visualisation per group was noticeable for most groups, by up to 28 percentage points (group D). Only group B had almost the same accuracy score for the map-like and nested treemap visualisations. The gap between the second and third highest scoring visualisation was even greater, ranging between 36 and 52 percentage points. Interestingly, despite these differences in accuracy, the average accuracy per group across the three types of visualisation evaluated was almost identical, ranging from 61% to 63%.

Within each group and for each image we counted the number of visualisation images with a 100% accuracy score, i.e. where each of the 10 participants in a group got the answer right. There were three times as many such perfect scores for the map-like visualisation than for the nested treemap (15 vs. 5 out of 40 scores per type of visualisation). The non-nested treemap did not have any scores of 100%, reaching as its highest a score of 90% which occurred only once. On the contrary, the non-nested treemap was the only visualisation that had a score of 0% for one of the images and groups, something that none of the other visualisations encountered.

We wondered whether inter-group differences in accuracy could be attributed to a learning effect, however the data did not support this assumption. Groups A and B first evaluated treemaps and then the map-like visualisation, whereas groups C and D evaluated the map-like visualisation first and then treemaps. If a learning effect had been present then the later evaluations would have benefited from the experience of having performed the earlier tasks. The same task performed later would thus have achieved a higher accuracy score than for those groups in which the order was the reverse. Observing the mean accuracy values between groups, however, we observe that this is not actually the case: groups C and D used the map-like visualisation first, whereas groups A and B used the map-like visualisation later. If groups A and B would have learned to perform the task better from their experience of having done similar tasks on treemaps, then the accuracy scores for map-like visualisations in groups A and B should be higher than in groups C and D. But the mean accuracy score of groups A and B combined is 82.5%, whereas the corresponding score for groups C and D combined is 88.5%. Comparing nested treemaps evaluated first or later we can see that only group B has evaluated nested treemaps first, but has the highest accuracy score (74%, the same as group C where it was evaluated last). In groups A and D nested treemaps were evaluated second, but score quite differently. Thus there does not appear to be a learning effect due to the order of evaluation of different visualisations.

To determine whether the difference in accuracy between the different kinds of visualisations was statistically significant we decided to look at performance on the level of individual participants, not at the group level. This is because we observed that there existed large variations in individual performance within a group. Thus we compared the performance in terms of accuracy of individual participants across the different visualisations. Figure 7 shows accuracy of each of the 40 participants (the first ten participants belonged to group A, the next ten to group B, and so on). We can notice a great amount of fluctuation: for non-nested treemaps it ranges through 70% (0%–70%), for nested treemaps the range is 100% (0%–100%), and for the map-like visualisation the range is 80% (20%–100%). We overlayed a polynomial trendline to smoothen these fluctuations and perceive a pattern. These trendlines again show the great distance between the non-nested and nested treemap accuracy figures, and a smaller but still clear distance between the nested treemap and map-like visualisation figures. The distance was smallest around participant 17, who belongs to group B, echoing our observation from the result of groups above. Detailed accuracy values of individual participants for each type of visualisation are shown in Table 12 in A.

We performed a two-tailed paired t-test on the accuracy figures, comparing pairs of visualisations that were neighbours in terms of accuracy. Additionally, an ANOVA test was performed to reassure the significant differences seen among different visualisations. The results are presented in Table 

7. The t-tests indicate that the difference in accuracy between non-nested and nested treemaps was highly significant ( close to zero), as was the difference between nested treemaps and map-like visualisations (). The ANOVA test shows a significant differences among the means of accuracy across three visualisations (). Given these results we can thus conclude that map-like visualisations indeed allow more accurate perception of hierarchy, which allows us to accept Hypothesis .

Figure 7: Accuracy by individual participants
Statistical Test -value
Non-nested vs. nested treemap 8.765E-14
Nested treemap vs. map-like 0.0008
ANOVA 1.461E-23
Table 7: Statistical significance of accuracy differences by individual participants

4.3 Speed

We measured the time taken to complete each evaluation task. The mean task completion times per group and per type of visualisation are summarised in Table 8. Figure 8 shows a plot of these mean task completion times. Counter to our expectation, we can observe that except in the case of group C, the map-like visualisation was the slowest type of visualisation to work with. Treemap was the fastest in all groups. In group B, the performance times of the two types of treemap were almost identical. In two of the other groups the nested treemap was clearly faster than the non-nested treemap, and only in group C was the nested treemap much slower than the non-nested treemap. Thus it seems that overall the nested treemap supported the fastest task performance.

Group Non-nested Nested Map-like Avg.
A 293 ( 54.8) 215 ( 26.9) 388 ( 106.3) 299
B 271 ( 49.9) 275 ( 78.0) 328 ( 75.2) 291
C 267 ( 76.4) 365 ( 61.9) 271 ( 66.5) 301
D 378 ( 92.5) 236 ( 50.6) 419 ( 86.6) 344
Avg. 302 273 352 309
Table 8: Mean speed of task performance by group for non-nested treemaps, nested treemaps and map-like visualisations (seconds) (ranges of the 95% confidence interval in parentheses)
Figure 8: Mean speed of task performance by group

To determine the significance of the speed differences we again looked at individual task performance times for all 40 participants. Figure 9 shows the speed values of each of the 40 participants. Again there is strong fluctuation across participants, so polynomial trendlines are again included in the chart. Detailed speed values of individual participants for each type of visualisation are shown in Table 13 in A.

We performed a two-tailed paired t-test and an ANOVA test on these individual speed figures. As the speed figures of different types of visualisations strongly overlap we analysed the significance for all three pairs of visualisations. The results are shown in Table 9. As the values show, the difference in speed between non-nested and nested treemaps was not statistically significant ( close to 0.2). For the difference between non-nested treemaps and map-like visualisations the difference was marginally significant ( close to 0.05). However, for the difference between nested treemaps and map-like visualisations the difference was more highly statistically significant (). That is, there is statistically significant evidence that map-like visualisations do not allow faster task completion than treemaps, at least for the type of task we evaluated and particularly when choosing a nested treemap. Also confirmed by the ANOVA test (), the mean speed of map-like visualisation was in fact higher than those two of treemaps. That is, map-like visualisations are slower than treemaps. Therefore we reject Hypothesis and instead accept the null Hypothesis .

Figure 9: Speed of task performance by individual participants
Statistical Test -value
Non-nested vs. nested treemap 0.1955
Non-nested vs. map-like 0.0402
Nested treemap vs. map-like 0.0048
ANOVA 0.0184
Table 9: Statistical significance of speed differences

4.4 Usability

This subsection presents the results of the measurements on two usability factors, namely the ease of understanding of the data presented in the visualisations, and the helpfulness of the visualisations for understanding the data shown in the visualisations.

4.4.1 Easiness to Understand

In the exit questionnaire, we asked the question: “Do you think it is easy to understand the data of the University through this visualisation?” to find out the perceived easiness to understand the dataset with each visualisation used in our experiment. Participants could respond in a 7-point Likert scale ranging from ”very difficult” to ”very easy” (where higher scores mean easier). Table 10 and Figure 10 shows the results of the easiness of understanding the data.

Group Non-nested Nested Map-like Avg.
A 4.1 5.7 4.5 4.8
B 3.7 3.6 4.6 4.0
C 4.0 5.2 4.5 4.6
D 3.9 4.5 3.9 4.1
Avg. 3.9 4.8 4.4 4.4
Table 10: Ease of understanding data through the visualisation, average values by group (values on a Likert scale: 1=very difficult, 7=very easy)
Figure 10: Ease of understanding data through the visualisation, average values by group (values on a Likert scale: 1=very difficult, 7=very easy)

As shown in Table 10, nested treemaps appeared easiest for users to understand the data, followed by the map-like visualisation; meanwhile non-nested treemaps scored the lowest in this area. However, by inspecting Figure 10, we discover that the measurement was fluctuating across different groups of participants. The ANOVA test further showed an insignificant result on the comparison among three visualisations (). As such, we cannot conclude that nested treemaps are better than both non-nested treemaps and map-like visualisations, in terms of the easiness for understanding the data. However, as suggested by the observation of our study, future research can look into the factors that make these visualisations perform differently in the ease-to-understand metric.

4.4.2 Helpfulness to Understand

Whether a visualisation is perceived helpfulness by the readers is another important usability measurement. In order to verify this, we asked another question: “Do you think this visualisation helps you to understand the data of the University?” in the exit questionnaire. This helped us to understand the overall helpfulness of these visualisations. Similarly, participants required to response to this question with a 7-point Likert scale, ranging from ”very unhelpful” to ”very helpful”. A higher score depicted a higher degree of helpfulness. Table 11 and Figure 11 list the results.

Group Non-nested Nested Map-like Avg.
A 3.9 5.6 4.5 4.7
B 3.9 4.1 4.6 4.2
C 4.3 5.4 4.1 4.6
D 3.5 4.6 3.9 4.0
Avg. 3.9 4.9 4.3 4.4
Table 11: Helpfulness of the visualisation for understanding the data, average values by group (values on a Likert scale: 1=very unhelpful, 7=very helpful)
Figure 11: Helpfulness of the visualisation for understanding the data, average values by group (values on a Likert scale: 1=very unhelpful, 7=very helpful)

Table 11 displays the mean values of helpfulness among all three visualisations. Nested treemaps were perceived the most helpful visualisation for understanding the dataset. Map-like visualisation came after nested treemaps, and non-nested treemaps were worst in our comparison. An ANOVA test showed a significant difference among all three visualisations (). In fact, the mean difference between nested treemaps and map-like visualisations are small (0.6 out of a 7-point scale). Therefore, we suggest that both nested treemaps and map-like visualisations are favourable for helping users to understand the data. Both visualisations are suitable for tasks that allow users to understand the data by themselves, in which such visualisations serve as a tool for helping users to navigate and perceive the data.

4.5 Qualitative Feedback

In the final section of the exit questionnaire, we asked the following open-ended questions for each type of visualisations, in order to capture additional qualitative feedback from the participants for their impressions on each visualisation:

“What do you think are the good or bad points about this type of visualisation?”

We received a mix of responses for each visualisation, which we discuss the main insights with representative quotes below. Grammatical and spelling mistakes in these quotes are preserved to maintain data integrity.

4.5.1 Non-nested Treemap

The following comments pinpointed the main weakness of non-nested treemap, which was the difficulty of understanding the hierarchy and the structure of the underlying data, as compared with nested treemaps. This was consistent with some prior work about visualising hierarchical data with treemaps Wijk1999; Liang2012; Tu2007. While colours and other visual aids might be helpful as suggested by the comments, the results suggested that non-nested treemaps were not the best option for visualising hierarchical data. Despite, many participants clearly stated that treemaps were good at demonstrating the size of the individual data item (which is an university department in our experiment).

“It’s hard to recognize, which is parent and which is child, because there’s no color or other thing to specific [sic] the relationship.” (P3)

“In terms of hierarchy, this creates confusion whether the pattern should be read horizontally or vertically, in or to understand the relationship from one to another.” (P4)

“I found it hard to compare two boxes across the image.” (P9)

“It is not clear to see the relationship between each part of the department but it clearly shows which department has more student.” (P25)

4.5.2 Nested Treemap

Participants agreed that nested treemaps addressed some of the problems found in non-nested treemaps. Nested treemaps were better for users to recognise the hierarchy and the structure of the underlying data. However, some participants commented that nested treemaps were sometimes difficult to understand, and required specialised knowledge to understand the visualisation. In this regard, we reckon that nested treemaps are preferred to non-nested treemap for showing hierarchical data, but the users’ capability for interpreting the visualisation needs to be considered.

“This type of visualization is better than the previous one but again as it is easier to ascertain the hierarchy and compare the dimensions. However, it still requires that the user has knowledge of this type of visualization and can interpret it.” (P5)

“This is a better pattern compared to the previous one, as it clearly shows the vertical relationships of the units.” (P4)

“For the top few lines it’s easy to understand; but for some parts at the bottom it’s a bit confused since some lines are not straight forward which makes me confused about which up-level should it belong to.” (P14)

4.6 Map-like Visualisation

For map-like visualisations, we received a majority of positive comments and few negative ones. Many participants expressed that it was easy to clearly understand the hierarchy and different levels of the data. In addition, the map metaphor was reassured by the feedback that the visualisation looked similar to a real geographic map. This further helps people to understand the underlying information, as discussed in our prior work PRMB2017; Pang:2016.

“The good point about this type of visualization is that the map-like visualization can be much easier to interpret when compared with the treemap.” (P15)

“The map-like visualization is by far (of all the presented) the easiest one to interpret as everything just seems (or appears) much simpler. One can easily determine which variables are at the same or different level.” (P5)

“It looks like a geographical map.” (P4)

“… similar to maps makes people more familiar with it.” (P30)

“To me, the information is easily read. And it more direct than a pie chart and more vivid than a Treemap.” (P21)

On the other hand, a minority of comments reported that the visualisation looked strange because it did not use the conventional visual representation (such as rectangles and circles) than other information visualisations, which made it hard to compare the size of different data items. Also, the positions of the text labels were crucial for describing the data hierarchy, and glitches in the visualisation software caused them to be misplaced. This needs to be addressed in future implementations to avoid misinterpretations of the data.

“It looks a bit strange as the irregular shape is very different from the conventional square or circle chart we use.” (P22)

“Furthermore, these types of visualizations don’t use numbers or percentages so there is no need to actually worry about comparing those, one simply has to look at the different visual dimensions. However, this can also prove to be a bad point as there is no way of determining the numbers, one may only infer.” (P5)

“The position of the name have to be precise, otherwise it may be troublesome to identify the relationship between elements/sectors.” (P35)

Overall, according to the feedback, the map-like visualisation shows a promising result of representing hierarchical data in an easily readable and understandable manner. It has a potential to be used as an alternative of both types of treemaps.

5 Conclusions

Over the past decade map-like visualisation has emerged as an attractive tool in the toolbox of the visualisation practitioner, one that has the advantage of ready understandability without prior training thanks to the wide exposure to maps in society. This makes it desirable to understand the strengths and potential weaknesses of this type of visualisation. In this paper we presented an evaluation that sought to test whether or not our assumption that map-like visualisations are more accurate and faster to use than treemaps is true.

The results and statistical analysis of our evaluation lead us to two main conclusions: (1) map-like visualisations are indeed better in terms of accuracy than both forms of treemaps we evaluated; (2) of the two types of treemap evaluated, the non-nested treemap performs very poorly in terms of accuracy. Our results therefore suggest that for tasks requiring the accurate recognition of hierarchy, a map-like visualisation should be preferred over a treemap; and that if a treemap must be employed for such tasks then the nested treemap should be greatly preferred over the non-nested treemap.

However, our results also showed that, at least in the experiment setup of our evaluation, map-like visualisations are slower than treemaps, even significantly slower in the case of comparing against nested treemaps. Nonetheless, in cases where speed is less important than accuracy, the use of map-like visualisations would still be beneficial. We note that the slower speed of the use of map-like visualisations warrants further study, perhaps exploring other types of map-like visualisation and other tasks.

Finally, in terms of statistical analysis of the usability factors, we cannot judge that any visualisation is easier for understanding the data than the others. However, nested treemaps are slightly more helpful than map-like visualisations for understanding the data, while non-nested treemaps perform very poor in this regard. Combining with the analysis of the qualitative feedback, we conclude that map-like visualisations is promising to be used as an alternative of treemaps, particularly where accuracy is required.

6 Acknowledgements

The support by the University of Macau Research Committee under grant number MYRG2014-00172-FST is gratefully acknowledged. We also acknowledge the help from Bin Pang for facilitating the experiment.

7 References


Appendix A Individual Survey Responses

The following tables show the detailed accuracy and speed values of all participants of our survey.

Participant Non-nested Nested Map-like
1 10% 70% 80%
2 10% 50% 100%
3 70% 100% 90%
4 20% 60% 90%
5 20% 80% 100%
6 40% 70% 80%
7 0% 90% 90%
8 0% 90% 100%
9 20% 50% 90%
10 20% 70% 80%
11 60% 40% 30%
12 40% 70% 90%
13 60% 100% 80%
14 50% 90% 80%
15 30% 80% 80%
16 20% 40% 40%
17 60% 80% 80%
18 10% 100% 100%
19 20% 70% 90%
20 30% 70% 80%
21 30% 90% 100%
22 20% 50% 70%
23 20% 40% 70%
24 40% 90% 100%
25 70% 80% 100%
26 30% 80% 100%
27 50% 100% 80%
28 20% 70% 100%
29 30% 80% 100%
30 10% 60% 20%
31 0% 0% 100%
32 30% 70% 80%
33 30% 100% 80%
34 30% 90% 100%
35 30% 90% 100%
36 20% 80% 100%
37 60% 70% 100%
38 10% 50% 100%
39 40% 60% 70%
40 30% 40% 100%
Table 12: Accuracy for non-nested treemaps, nested treemaps and map-like visualisations by individual participants
ID Non-nested treemap Nested treemap Map-like
1 254 180 499
2 332 269 655
3 202 159 270
4 278 194 239
5 215 147 439
6 213 209 250
7 361 265 449
8 458 251 618
9 393 236 327
10 227 237 138
11 233 159 211
12 185 177 301
13 190 164 356
14 323 358 463
15 404 257 251
16 226 166 286
17 335 493 472
18 276 368 219
19 360 428 530
20 179 181 193
21 156 370 226
22 273 507 395
23 496 465 242
24 327 431 279
25 451 349 521
26 200 437 195
27 156 283 165
28 240 225 214
29 224 373 241
30 144 212 231
31 349 43 285
32 277 168 246
33 477 297 414
34 438 290 353
35 368 207 259
36 359 297 531
37 683 243 519
38 143 279 690
39 232 232 453
40 456 306 440
Table 13: Speed of task performance for non-nested treemaps, nested treemaps and map-like visualisations by individual participants (seconds)
ID Non-nested treemap Nested treemap Map-like
1 1 5 1
2 5 7 7
3 1 6 4
4 6 5 2
5 2 3 5
6 6 6 5
7 4 6 5
8 6 6 6
9 5 6 4
10 5 7 6
11 1 4 5
12 5 6 4
13 6 5 3
14 2 2 7
15 5 2 7
16 2 2 2
17 7 1 6
18 4 7 7
19 2 4 1
20 3 3 4
21 3 4 5
22 7 7 2
23 5 5 4
24 2 6 4
25 2 5 3
26 1 4 3
27 2 6 6
28 7 2 7
29 6 6 7
30 5 7 4
31 5 4 5
32 3 2 1
33 5 6 3
34 5 5 3
35 4 3 5
36 4 4 4
37 6 7 6
38 1 3 6
39 1 7 4
40 5 4 2
Table 14: Perceived ease-of-use for non-nested treemaps, nested treemaps and map-like visualisations by individual participants (7-point Likert scale)
ID Non-nested treemap Nested treemap Map-like
1 1 5 1
2 6 7 7
3 1 6 4
4 5 6 3
5 2 3 5
6 7 6 5
7 4 5 5
8 5 6 6
9 5 5 3
10 3 7 6
11 2 4 5
12 5 5 4
13 4 5 3
14 1 5 7
15 6 3 6
16 5 5 3
17 7 1 6
18 4 6 7
19 2 4 1
20 3 3 4
21 4 5 6
22 7 7 2
23 5 4 4
24 3 6 4
25 2 6 2
26 2 4 3
27 2 6 6
28 7 3 5
29 6 6 4
30 5 7 5
31 3 4 4
32 4 4 1
33 5 7 4
34 4 5 3
35 4 4 7
36 1 1 2
37 6 6 6
38 2 4 6
39 1 7 4
40 5 4 2
Table 15: Perceived helpfulness for non-nested treemaps, nested treemaps and map-like visualisations by individual participants (7-point Likert scale)