What is "Intelligent" in Intelligent User Interfaces? A Meta-Analysis of 25 Years of IUI

03/06/2020 ∙ by Sarah Theres Völkel, et al. ∙ Universität München 0

This reflection paper takes the 25th IUI conference milestone as an opportunity to analyse in detail the understanding of intelligence in the community: Despite the focus on intelligent UIs, it has remained elusive what exactly renders an interactive system or user interface "intelligent", also in the fields of HCI and AI at large. We follow a bottom-up approach to analyse the emergent meaning of intelligence in the IUI community: In particular, we apply text analysis to extract all occurrences of "intelligent" in all IUI proceedings. We manually review these with regard to three main questions: 1) What is deemed intelligent? 2) How (else) is it characterised? and 3) What capabilities are attributed to an intelligent entity? We discuss the community's emerging implicit perspective on characteristics of intelligence in intelligent user interfaces and conclude with ideas for stating one's own understanding of intelligence more explicitly.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Intelligent User Interfaces

are the eponymous subject of study at the IUI conference, which celebrates its 25th anniversary this year. Interestingly, it still remains difficult to define what exactly renders an interactive system or user interface “intelligent”, not only at IUI but also in the intersecting fields of Human-Computer Interaction (HCI) and Artificial Intelligence (AI) at large. This shows both in a lack of widely-referenced or recent objective definitions and also our subjective experiences, for example, in workshop discussions with fellow IUI researchers on the topic.

If we have managed so far, should we bother with such terminological considerations? We argue that our understanding of intelligence in UIs can implicitly shape our work, for example, by influencing our assumptions, methods, and metrics, and not least simply by what we consider to be in-scope of IUI work. As highlighted in similar reflections [34], our understanding of such a core concept thus serves as a thinking tool for our research. It seems adequate to examine such a tool from time to time to surface and make explicit its assumptions, usages, and boundaries.

Explicating our collective understandings of intelligence in UIs also serves as a communication tool: Developing conceptual clarity contributes to a common belief system and terminology which is crucial for fostering productive discussions and exchanges, in particular for interdisciplinary research communities, such as IUI.

The literature demonstrates that such reflections trigger important discussions: For example, Kostakos [37] highlighted the “big hole” in HCI research, based on a co-word analysis of 20 years of CHI conference proceedings [44]. The main critique therein is a lack of “motor themes”, which are both central to a community and dense (internally coherent). In other words, HCI appears to lack core themes sufficiently developed to become mainstream – as opposed to jumping between novel themes from year to year. CHI is a broad HCI conference. In contrast, IUI seems to directly carry such a motor theme in its name and focus – intelligent user interfaces. Therefore, taking intelligent UIs as a central theme in the IUI community, we examine whether it is also dense, in the sense of being coherent in its understanding of the underlying core concept.

In particular, in this reflection paper, we take the 25-year milestone as an opportunity to analyse in detail the meaning of “intelligent” in the IUI community: To do so, we applied text analysis to extract all occurrences of “intelligent” in all IUI proceedings. We then manually reviewed these with regard to three main questions: 1) What is deemed intelligent? 2) How (else) is it characterised? and 3) What capabilities are attributed to an intelligent entity?

Our results show a growing variety over the years of what is referred to as intelligent and how it is characterised as such. At the core, IUI researchers tend to describe intelligence in their work with the key aspects of automation, adaptation, and interaction. Researchers also associate different descriptors with intelligent technology depending on how it is conceptualised (e.g. as a UI, system, or agent). Moreover, we extract nine challenges that researchers mention in the context of intelligent entities.

2 Background

2.1 Large-Scale Literature Text Analysis

Recent work has brought forth examples of literature reviews based on large-scale text analysis: For example, Liu et al. [44]

analysed co-occurrences of the “author keywords” sections in 20 years of CHI papers. Using clustering analysis, they derived core, popular, and backbone topics, and examined keyword networks. They discussed trends between two periods and the emerging structure of HCI research, revealing that it lacks “motor themes” that are both central and dense. Similarly, Lee et al. 

[41] analysed the citation network of CHI papers and investigated keywords in the abstracts to understand long-term topic trends. Their work identified emerging research topics and showed the rise and fall of different topics over the past three decades of CHI. As mentioned in the introduction, these approaches motivate our analysis of a more specific venue with a stated focus, that is, intelligent user interfaces. In contrast to Liu et al. we analyse the papers’ full text for a specific key term (“intelligen*”, i.e. intelligen-t/ce/tly etc.). We then also study co-occurrences of this key term with other descriptions (e.g. an entity might be called both “intelligent” and “adaptive”). This allows us to gain insights into the researchers’ implicit understanding of what is considered intelligent.

In another large-scale literature review, Abdul et al. [2]

recently used topic modelling to analyse over 12,000 papers from HCI and Machine Learning (ML)/AI regarding aspects such as explainability of intelligent interactive systems. Their resulting topic networks gave insights into the connections between research fields. As a key implication, the authors called for more interdisciplinary research at the intersection of HCI and AI to address the emerging challenges. Since IUI is placed at precisely this intersection, this motivates our detailed analysis of IUI proceedings in this paper. In particular, we focus on the emerging understanding of the core concept of intelligent systems and user interfaces in this interdisciplinary research.

Similar approaches have also been applied within the IUI community to analyse not the literature but the users’ views on interactive intelligent systems: Eiband et al. [19] applied topic modelling to 35,448 reviews of apps with algorithmic decision-making on the Google Play Store. Combined with manual analysis, they revealed the problems that users faced during interaction. In contrast, our work focuses on the perspectives of researchers on intelligent user interfaces and systems.

In another recent literature reflection, Hornbæk et al. [33] analysed 35 years of CHI proceedings regarding the use and understanding of the term “interaction”. They categorised 2,668 distinct terms used to modify and further describe interaction. Their results show growing diversity in such characterisations and that these styles of interaction play an increasing role in discourse in the field. In a similar manner, in this paper, we examine the use and description of the key term “intelligen*” in IUI proceedings.

2.2 What is Intelligence?

Intelligence is an intrinsic human characteristic, and as such, has been transferred to computer science. This transfer shows, for example, in Turing’s article from 1950, where he asked: “Can machines think?” [72], and which can be considered the beginning of the discussion on the nature of technological intelligence. However, finding a common understanding and conceptualisation of human intelligence in the first place has been the subject of intense scientific discussion in the past. To date, there is no single definition, which may partly account for the current diversity of notions in our field.

In the following, we first give a short overview of exemplary definitions of human intelligence as presented in psychology, before we compare them to example definitions of technological intelligence.

2.2.1 Human Intelligence

In 1921, the discussion about human intelligence was the subject of the symposium of Educational Psychology on “What do I conceive intelligence to be?”. A number of experts in the field shared their views on the question, arguing that, for example, thinking abstractly or having the ability to learn and respond to questions are essential for intelligence (cf. [40, 65]).

In later years, the capability to adapt to the environment has been introduced as another central part of intelligence. For example, Sternberg [66] argues that “intelligence comprises the mental abilities necessary for adaptation to […] any environmental context […] It offers people an opportunity to respond flexibly to challenging situations” [66]. The capability of adaptation to new contexts is also emphasised by the American Psychology Association, describing intelligence as “[…] the ability to derive information, learn from experience, adapt to the environment, understand, and correctly utilize thought and reason” [1].

Besides trying to find general prerequisites for intelligence, the concept is often broken down into sub-types. For example, Sternberg defines intelligence via emotional, practical, analytical, and creative intelligence [66]. Gardner even suggests that the brain consists of eight different forms of intelligence operating in autonomy, namely linguistic, logical-mathematical, spatial, musical, physical-kinaesthetic, interpersonal, and intrapersonal intelligence [26].

Within each individual human being, the manifestation of these different sub-types of intelligence can vary [26, 27]. This raised the question if intelligence should be seen as a holistic concept, or if it is rather certain capabilities that form intelligence (i.e., mathematical or linguistic skills).

2.2.2 Technological Intelligence

Especially early work involved in establishing IUI as a research area reflects on what it means for technology to be intelligent. On closer inspection, they take up different aspects from the definitions presented above.

For example, Stephanidis et al. [64] in 1997 characterised IUIs “[…] by their capability to adapt at run-time and make several communication decisions concerning ‘what’, ‘when’, ‘why’ and ‘how’ to communicate.” Therein, they relate to an earlier view of a UI communicating concepts to the user [68]. They thus particularly highlight adaptation and decision-making, which in part resounds with Sternberg’s [66] definition and the one given by the American Psychology Association. Moreover, knowing “what”, “when”, “why” and “how” to communicate can be considered to involve some sort of interpersonal intelligence.

Similarly, in 2000, Höök [32] reflected on “Steps to take before intelligent user interfaces become real”: In her account, an important emerging view of IUIs was adaptation and personalisation. Another related aspect is generating UIs to automatically adapt to varying contexts, such as the device. This prominently appeared in 2004 in the SUPPLE system by Gajos and Weld [25].

Moreover, Wahlster and Maybury [74] stated that intelligent user interfaces “are human-machine interfaces that aim to improve the efficiency, effectiveness, and naturalness of human-machine interaction by representing, reasoning, and acting on models of the user, domain, task, discourse, and media (e.g. graphics, natural language, gesture).” Their definition entails the ability to think and to understand language and gestures (linguistic and physical-kinaesthetic intelligence). Also, it defines the goal or purpose of technological intelligence – efficiency, effectiveness, and naturalness of interaction.

These exemplary definitions give an idea of how technological intelligence is inspired by human intelligence or capabilities. Beyond that, it is shaped through the interaction context with users towards fulfilling a particular purpose. Yet, these aspects often remain tacit in the literature.

With our work, we aim to shed light on what (parts of) technology researchers attribute intelligence to in the IUI community (besides the eponymous term user interface), how this intelligence is characterised, and which capabilities are ascribed to the intelligent entities. We return to the listed definitions and reflections in our discussion at the end of this paper after presenting the results of our analysis.

3 Method

With our approach, we aim to gain deeper insight into the understanding of the term “intelligen*” as used in the IUI community. Thus, we analyse the utilisation of this notion throughout the past IUI conference proceedings. We follow the approach by Hornbæk et al. [33], who extracted occurrences of the term “interaction” from the proceedings of 35 years of CHI. In our case, we extract occurrences of “intelligen*” from all IUI papers. Similar to their work, we then analyse these extractions and interpret themes and trends.

3.1 Data Acquisition and Extraction

We obtained all papers published during the last 24 IUI conferences from 1993 until 2019 (note that there were no IUI conferences from 1994–1996) through a request to the ACM Digital Library. IUI as a conference has changed over two and a half decades and so have the proceedings. Starting out as a small conference with 39 accepted papers in total in 1993, the IUI main proceedings comprised 71 full and short papers as well as adjunct proceedings including posters, tutorials, and workshops in 2019. For our analysis, we only included full and short peer-reviewed papers. To compile this set, we manually went through all IUI proceedings in the ACM digital library111https://dl.acm.org/conference/iui, accessed 17.01.2020. This resulted in a data set of 1,111 papers. Please note that since we included both long and short papers, the number of accepted papers does not correspond with ACM’s acceptance rates, which often only refer to long papers. For these papers, we then removed all meta information222In particular, we removed title, author, affiliations, conference name, theme, data, location, session title, keywords, references, copyright statement. so that only the text body was considered for our analysis. We automatically extracted all sentences from the data set in which the term intelligent or its grammatical derivatives (intelligence, intelligential, intelligently) occurred at least once. We thereby acquired 1,804 sentences.

3.2 Data Analysis

Due to the number and extent of CHI full texts, Hornbæk et al. [33] relied on sentence parsing, which is limited in terms of the semantic detail that can be automatically captured. In contrast, the IUI dataset is small enough to allow us to combine a large-scale automated text analysis with an in-depth manual content assessment. Therefore, we could not only infer syntactical information on the usage of “intelligent” but also semantic information extracted from the context of the sentence.

In particular, our analysis focuses on three central aspects of the sentences:

  1. What is considered “intelligent”? [entity]

  2. Which other descriptors are used to further specify this intelligent entity? [co-descriptor]

  3. What actions / capabilities are attributed to this intelligent entity? [action]

At first, all authors reviewed the 2,011 sentences to develop a code book for entities, co-descriptors, and actions. We then selected a random sample of four sentences from each year’s proceedings (resulting in 96 sentences), which were coded by four coders independently using the predefined coding rules. We compared the results and discussed all inconsistencies until we reached a consensus. Afterwards, we split the dataset among four coders for the final coding. In particular, we established the following coding procedures:

Entity: We define the intelligent entity as any object, which is ascribed intelligence in the respective sentence. Hence, entities usually comprise one noun or a series of nouns, such as “user interface”, “system”, or “information collection system”. For our analysis, we identified and labelled the entity based on this noun / series of nouns. After checking the use of interface in the coded sentences, we merged user interface and interface into one entity.

Co-descriptors: Co-descriptors either refer to the intelligent entity (e.g. “user-adaptive interactive systems” [15]) or the described action (e.g. “automatically extract human gestures” [71]) to further specify it. Here, we coded the infinite form of the adjectives as co-descriptors (e.g. user-adaptive, interactive and automatic for the examples above). In a few cases, we merged multiple variations into one co-descriptor (e.g. automated, automatic, autonomous; and context-sensitive, context-dependent, context-aware).

Action: The action describes what the intelligent entity does, i.e. its capabilities. We first manually identified the part of each sentence that refers to the action of the intelligent entity. These sentence parts could not be meaningfully counted when stated directly as they appear in the text (i.e. almost all occurred only once). Hence, we clustered them along their key verb (e.g. “support […] information seeking” [67] to support). Then, four coders discussed and merged clusters of verbs based on synonyms in this context (e.g. aid and help, collect and gather) and, similar to the process applied to entities and co-descriptors, variations of British English and American English (e.g. analyse and analyze). We thereby obtained 60 distinct actions. Finally, we manually checked all resulting actions and corresponding codes and excluded single codes which did not fit the action verb cluster. Similarly, we went through the list of remaining codes to assign them to one of the key action verbs if possible.

4 Results

4.1 Dataset Overview

In total, we analysed 1,804 sentences containing our search terms. These come from 504 papers (out of 1,111 accepted papers) over 24 IUI proceedings (1993 - 2019). Each year on average contributed 75.17 sentences () from 21.00 papers (). Contributing papers on average contributed 3.58 sentences ().

Table 1 shows summary statistics for our main coding categories (entities, co-descriptors, actions). Figure 1 visualises the number of coded sentences and papers over time. The figure also includes the total number of accepted papers per year333https://sigchi.org/conferences/conference-history/iui/, accessed 17.01.2020, showing that not every paper uses “intelligen*” (i.e. compare accepted papers and contributing papers).

entities co-descriptors actions
codings 2034 673 1308
 mean (SD) per paper 4.22 (5.81) 2.54 (2.48) 3.79 (3.71)
distinct codes 522 250 59
 mean (SD) per paper 2.50 (2.14) 2.03 (1.53) 2.99 (2.37)
papers (% of all) 483 (95.83%) 266 (52.78%) 346 (68.65%)
Table 1: Overview of our coding. From top to bottom, rows show: number of codings per category (and mean / SD per analysed paper), number of distinct codes per category (and mean / SD), and number of papers that contributed at least one coding (percent of all contributing papers).
Figure 1: Overview of the coded data over time, in terms of absolute counts of coded sentences and papers (i.e. papers which include “intelligen*” at least once). The figure also shows the total number of accepted papers at each conference.

4.2 Top Entities, Co-Descriptors, and Actions

Figure 2 shows the histograms of the top codes per category.

4.2.1 Entities

Our analysis yielded 2,034 codings of 522 distinct entities. 95.83% of papers including the term “intelligen*” mentioned at least one intelligent entity with a mean of 2.50 () distinct entities per paper (cf. Table 1).

On the one hand, the entities which researchers labelled as intelligent comprise different kinds of technology (e.g. “system”, “tool”, “agent”) or components thereof (“interface”, “technique”, “algorithm”). On the other hand, they comprise a variety of functionalities or features (“assistance”, “selection”, “support”). The identified entities cover both intelligence in the interface versus intelligence internally in the system, as also discussed by Norman in 2002 [51].

The majority of coded entities (81%) were only mentioned once due to their specificity, such as “search support function” [47] or “ToDo manager” [58]. On the other end, five entities were reported in twenty or more papers.

The eponymous intelligent user interface is the most frequently mentioned entity, which is included in 210 papers (cf. Figure 2). Another top entity is the intelligent system (115 papers). Moreover, the entities agent and assistant are mentioned in 61 and 28 papers, respectively. Other potentially expected intelligent entities such as recommender system and robot were only directly described as intelligent in very few papers.

Looking at the remaining entities beyond the top list in Figure 2, we found, for example, further 62 distinct “variations” of system with a total paper occurrence count of 76: For example, papers mention expert system (3 papers), help system (3), interface system (3), and dialogue system (3). All others were mentioned in fewer than three papers. Similarly, we found further 39 distinct “variations” of interface with a total paper occurrence count of 43: These tend to refer to specific application contexts such as acquisition interface (2 papers), visualization interface (2), or cinematography interface (1). As another example of such specific applications, recommender systems had ten further “variations” with a total paper occurrence count of eleven, including product recommender and presentation recommender.

Figure 2: Overview of the top 25 occurring codes, ranked by number of papers in which they occur over all years.

4.2.2 Co-Descriptors

Researchers used a variety of co-descriptors to further describe characteristics of intelligent entities. We found 250 distinct codes of these co-descriptors, which occurred 673 times in total. 52.78% of the papers contributed to this list of co-descriptors with a mean of 2.03 distinct codings per paper ().

Again, the majority of co-descriptors occurred only occasionally with eight co-descriptors mentioned in ten or more papers. The most frequently used co-descriptor is autonomous (53 papers). For example, in 1993, Woods characterised an intelligent interface as “an autonomous computer agent who mediates between process and practitioner” [76]. The co-descriptor adaptive was featured in 28 papers, for example, in 2004: “The intelligence of the adaptive interfaces resides in the migration server, which adapts data collected at runtime […]”  [6]. In 20 papers, intelligent entities were also labelled interactive, for example, in 2013: “These techniques raise major challenges for interactive and intelligent systems design to support the consumers’ understanding and control of their energy usage in this complex environment.” [23]. Another top co-descriptor is dynamic. For example, in 2011 Bourke et al. [11] stated “that the time is now right for more dynamic and intelligent interfaces to play an important role in a new generation of connected cameras”. Additionally, in thirteen papers designing natural intelligent user interfaces was declared an important goal. For example, in 1997, Birnbaum et al. noted: “Ironically, people often apply AI to interfaces to make them more natural, but without crafting the interface to support the intelligence, the AI component makes the interface far less usable” [8].

4.2.3 Actions

In 68.65% of papers, researchers described actions that intelligent entities perform. We coded 1,308 occurrences of such action descriptions, which we grouped into 60 key actions as described above. 187 codes could not be assigned to any cluster but describe individual cases, which are excluded for the following analysis. On average, we found a mean of 2.99 distinct action codes per paper ().

In 93 papers, an intelligent entity was used to assist, aid or help users, e.g. in “performing their tasks” [9], “understanding and navigating information” [78] or improving a variety of skills, such as for example “develop writing proficiency” [9] or improve “public speaking” [70]. Moreover, the intelligence can support the user by guiding, facilitating, or easing a task, e.g. assisting “user orientation” [57], “comprehending an information space” [4], or “understanding why a system decision has been made” [19]. On a more functional level, intelligent entities create or generate a huge variety of functions, content, or models, for example, “simulations” [35], “explanations” (e.g. [12]), “appropriate content for display” [30], etc.

Intelligent entities are also concerned with detecting or capturing specific information, for example, to “recognize the user’s emotional state” [10] or to “diagnose errors or misconceptions” [20].

Figure 3: Development of total number of distinct entities, co-descriptors, and actions over time. Read: At each point, Y distinct codes have appeared until and including year X. This figure shows a growing diversity in what IUI researchers refer to as intelligent and how they describe it.

In 37 papers, the task of intelligent entities was generally described as interacting with the user (e.g. “interact with learner” [56]

) or parts of the entity (e.g. “interacting intelligently with the natural language generation components” 

[45]). Furthermore, several papers specified how this intelligent interaction should be, for example, “interact in a natural, more engaging way” [56].

An important role of intelligent entities also seems to be to adapt. On the one hand, papers describe to whom or what the intelligent entity adapts, e.g. “adapt to [the user] in real time” [43] in general or specific user features (e.g. “adapt [the agent’s] behavior to that of the [user’s] personality” [77]). On the other hand, papers provide more information about which features of the intelligent entity are adapted, e.g. “adapt interface” [6] or “adapt data in runtime” [6].

4.3 Trends over Time

We also analysed the occurrence of these entities, co-descriptors and actions over time. The main overall trend is an increasing variety of what is referred to as intelligent and how it is described, as visualised in Figure 3: Each IUI conference added new entities and co-descriptors to the overall set, while the main actions were present already from very early on.

We found no clear trends regarding the numbers of (new) distinct counts per year. We further examined the development of specific entities, co-descriptors, and actions over the years but also found no emerging trend patterns.

4.4 Co-Occurring Codes

Figure 4: Co-occurrences of the top coded entities and co-descriptors. Colour shows the relative occurrence of co-descriptors per entity. The dendrogram shows the clustering.
Figure 5: Co-occurrences of the top coded entities and actions. Colour shows the relative occurrence of actions per entity. The dendrogram shows the clustering.

We further analysed how often different codes occur together: In particular, we counted 1) co-occurrences of entities and co-descriptors, and 2) co-occurrences of entities and actions.

For instance, the sentence “Evaluating the impact of expertise and route knowledge on task performance can guide the design of intelligent and adaptive navigation interfaces.” [52] contains both an entity (navigation interface) and a co-descriptor (adaptive).

For presentation and visual clarity, we applied clustering to arrange rows and columns of a co-occurrence clustermap. Informed by related work using clustering in literature analysis [44]

we chose simple agglomerative (bottom-up) hierarchical clustering. Concretely, we used the

seaborn444https://seaborn.pydata.org/generated/seaborn.clustermap.html, accessed 17.01.2020 library for Python with the UPGMA algorithm [62] and default settings.

Figure 5 shows such a clustermap for entities and co-descriptors. Note that the shown values are relative to each entity (i.e. co-occurrence counts in each row are divided by the row sum). This allows for a better visual comparison between entities. Complementary, Figure 5 shows the clustermap for entities and actions. Overall, these figures illustrate the variation between intelligent entities with regard to the co-descriptors and actions that researchers associate with them.

4.5 Descriptions of Intelligence

We also analysed descriptions of intelligence as a concept itself (i.e. “intelligence” as a noun). This occurs rather sparsely, 134 times in total. The top descriptors of intelligence in our data are artificial (60 times), machine (11), social (10), ambient (6), computational (5), emotional (5), and explainable (5). Further 21 distinct descriptors of intelligence occurred less than five times each.

4.6 Challenges in Designing Intelligent Technology

Designing intelligent technology is challenging from a technical and a human-computer interaction perspective. We manually extracted such challenges as indicated in our data set through keywords like “problem”, “issue”, “need”, and so on. In this way, we collected 117 occurrences of challenges which we clustered and summarised into nine overarching themes. We next describe these themes along with references to example papers.

4.6.1 Presentation in the Interface

Several papers named the presentation of technological intelligence in the interface as a challenge. This refers to specific aspects, such as whether and how to present uncertainty [23], but also generally highlights the interface as a limited resource that needs to be managed carefully [53].

4.6.2 Negative Effects of Intelligence

Our analysis revealed that researchers embrace the potential benefits of technological intelligence, but also caution against possible negative side effects with regard to users. Most often stated were user frustration due to a lack of algorithmic accuracy [73, 75], and potentially tedious interaction to influence the algorithm [46]. Höök [31] even argues that technological intelligence may hinder the development of clear mental models of the system and thus negatively impact interaction.

4.6.3 Explanation

A further challenge prevalent in our data set is the adequate explanation of system behaviour to support understanding of system suggestions, predictions, and decisions. From an HCI perspective, this entails assessing what users want and need to have explained [54]. Mynatt and Tullio [49] highlight that fostering user understanding of technological intelligence becomes particularly important in wearable and ubiquitous computing.

4.6.4 User Interaction

Interaction with the user was repeatedly named as a crucial and multi-faceted challenge for designing intelligent technology. As Smith and Lieberman put it, “people interact with interfaces to accomplish goals […]” [61]. Along these lines, interaction requires establishing awareness of a shared context, reasoning about shared goals, planning and allocating resources to reach these goals, as well as communicating and evaluating progress [3].

This includes creating a suitable way to communicate about goals [3], providing feedback options [29], and supporting error detection and correction  [48, 63]. However, interaction needs “balance between user control and autonomy […] in a flexible way without overwhelming the user with requests, and without risking undesired system actions” [23].

4.6.5 Technical Challenges

The technical challenges involved in designing intelligent entities are manifold. With regard to training a learning system, difficulties referred to limited training data in early stages of use [75] and finding patterns in the data [13]. A closely related challenge is the management of false positives and recognition errors against the negative cost of reject rates [36]. More generally, technological intelligence was linked to the control of complex environments [23], to the interaction via multiple channels or devices [24, 48], and to ensuring system robustness [48]. Moreover, capturing user data such as their affective state [69] or interpreting imprecise natural language instructions [42] was named as technically challenging.

4.6.6 When to Integrate Intelligence

In the papers we analysed, researchers called for a considerate integration of technological intelligence. That is, a core challenge for designing intelligent technology is to reflect on when to make technology intelligent in general [17], to decide on where to integrate intelligence [8] and for what kind of tasks [55]. These considerations need to balance the advantages of artificial intelligence techniques against those of traditional methods, and the system complexity against user benefits [8]. This is in line with Nagel [50], who argues that a challenge for designing IUIs is to prove that intelligent behaviour (e.g. adaptation) actually improves interaction.

4.6.7 Trust, Confidence, Reliance and Privacy

A central challenge prevalent in our data set was to establish user trust in the technology and confidence in its outputs (cf. [22, 39]), to demonstrate the trustworthiness of the technology [23], and to foster appropriate reliance [28]. Moreover, privacy issues (e.g. through camera-based user tracking) were mentioned as a concern [5].

4.6.8 Usability and Usefulness

Usability, usability issues, and the violation of usability principles were named by many papers in our data set (e.g. [8, 28, 75]) and decreased productivity and performance as a related concern [38, 75]. This also comprises the costs involved in verifying and correcting mistakes by intelligent technology, such as false inferences [8]. Furthermore, in order to be useful, intelligent technology should consider users’ changing tasks and needs [7].

4.6.9 Evaluation and Standards

Finally, the analysed papers called for evaluation criteria (e.g. for visualisation techniques [16]) and standards, for example for transparency [18] and interface design [48].

5 Discussion

5.1 Limitations

Investigating the notion of “intelligence” in IUI through automated text analysis of how it is used in the written proceedings comes with a set of limitations:

The ACM Digital Library provided us with xml files for all IUI conference proceedings, which included the full text and meta information for each paper. These full texts were likely generated by automatically extracting text from pdfs. Hence, single information or texts were missing and parsing errors occurred (e.g. not recognising “fi” as two separate characters). We addressed this challenge by manually searching for missing parts. However, we cannot rule out that occasional occurrences of “intelligen*” could not be detected by our automatic analysis.

Furthermore, we extracted single sentences that include “intelligen*”. Hence, the context of the paragraph is not considered, potentially missing further definitions or specifications of the term. For example, indirect mentions through grammatical placeholders or antecedents in subsequent sentences could reveal more information on the characteristics of “intelligent” entities (e.g. “We implemented an intelligent system. It is characterised by […].”). The same applies to the identified challenges, which should not be seen as a comprehensive overview of all challenges discussed at IUI. Rather, we can think of them as those challenges typically mentioned when describing something as intelligent.

Examining the entire paragraph rather than only the sentence for each occurrence of “intelligen*” could also give further insights into how deeply the notion of intelligence is considered in the individual papers. For example, the use of “intelligen*” could either be limited to a single sentence, e.g. to label a system or be embedded in a paragraph of deeper discussion.

Moreover, for the scope of this work, we did not include synonyms of intelligent, such as “smart”, or abbreviations (e.g. “AI” – Artificial Intelligence, or “ITS” – Intelligent Tutoring System).

We favoured greater semantic detail through manual analysis of the extracted sentences over further automated text analysis on a larger data set (e.g. including further proceedings such as CHI; cf. the interaction analysis of Hornbæk et al. [33]). Yet, our analysis only covers explicit evidence of the use of “intelligen*” in IUI papers and assigning them to a set of themes. Thus, implicit meanings and conceptions of intelligence may not be depicted in these themes. For example, an understanding of intelligence could also be derived from the characteristics of systems that were presented in IUI papers. Since such an analysis would require an in-depth reading of all articles, it was out of scope for this paper.

5.2 Adaptation, Automation, and Interaction are at the Core

Our analysis shows that adaptation, automation, and interaction are the most common aspects that IUI researchers highlight when describing something as intelligent. These co-descriptors are present across all years. Based on our analysis, we view them as the most coherent core of intelligence in IUI work.

These findings align with early reflection on IUIs: For example, Höök [32] also emphasised adaptation and personalisation in her account of IUIs in 2000. Similarly, Stephanidis et al. [64] early on highlighted that IUIs adapt the way they “communicate” with the user. Moreover, Wahlster’s [74] definition of IUIs in 1998 (see Background Section) starts its list of descriptors for IUIs with improving efficiency and effectiveness, which also often seems to underlie the motivation for adaptation and automation. Regarding their combination, both adaptation and automation are, for example, married prominently in one of the most widely referenced IUI papers: SUPPLE [25] was a system for automatically generating UIs to adapt to different devices and usage patterns. Finally, interaction arguably underlies the HCI side of IUI in general throughout all such work.

While automating a process is also included in the most frequently mentioned actions (e.g. creating, detecting, recommending), our list of actions underlines the importance of IUIs to primarily assist the user. These findings indicate that in the discussion of the role of artificial intelligence to replace or augment humans, the IUI community emphasises the importance of assisting or empowering humans rather than replacing user skills.

In conclusion, adaptation, automation, and interaction provide a common denominator for what many researchers regard as intelligent about IUIs. There are, of course, countless different ways that researchers envision this to unfold, for instance, concerning how automation is integrated into an interactive system.

5.3 IUI Embraces both (Tool-) UIs and Agents

A key aspect in Höök’s [32] early reflection about IUIs concerned tools and systems vs. agents: “Here we must be careful of imitating human-human communication and assuming that that would be the best model of performance.” This echoes the longstanding wider discussion of whether HCI should embrace intelligent technology as agents or as tools. Most prominently, these views have been debated by Shneiderman and Maes (e.g. in 1997 [59]), and this is still ongoing and relevant twenty years later [21].

In our analysis, we indeed see this fork in the path appear for the IUI community as well: One the one hand, researchers describe user interfaces, systems, and tools as intelligent. On the other hand, we see descriptions of intelligent agents and assistants. This reveals that IUI has broadly embraced both intelligence for more traditional UIs (e.g. GUIs), as well as for agents, such as chatbots and voice agents.

Our analysis reveals diverging tendencies of what intelligence entails for such different entities: For example, intelligent interfaces and systems tend to be described relatively more as adaptive and interactive, compared to agents and assistants. These, in turn, tend to be described relatively more as autonomous. Assistants are also often labelled as personal. This suggests that researchers’ implied assumptions and goals around intelligence vary depending on whether the technology is seen, for example, as a UI, system, tool, or agent.

5.4 Examining Emerging and Explicit Descriptions of Intelligence

Equipped with the terms of our analysis, we can also critically examine how these terms are related to definitions of intelligence: For example, taking a definition by Singh [60], a system is intelligent if we need to “attribute cognitive concepts such as intentions and beliefs to it in order to characterize, understand, analyze, or predict its behavior”.

Therein, our most common descriptors seem to be reflected as follows: Adaptation can be connected well to changing behaviour, for example, based on changing “beliefs” of the system about the state of the world (e.g. the user’s context for context-aware UIs). In addition, automation brings a certain independence from human actions, which could be seen as implying own intentions and beliefs (e.g. in voice assistants that perform tasks for the user).

Moreover, comparing the identified actions of intelligent entities with the understanding of human intelligence (see Section 2.2.1), it is interesting to note that apart from the ability to think abstractly [40, 65], the other capabilities associated with human intelligence – such as learning [40, 65], responding to questions, adapting to the environment [66], understanding [40, 65] – are among the most frequently mentioned actions.

In this way, the revealed descriptors and actions can help us to examine previous and future definitions of intelligent technology in the light of the emerging views from the literature (and vice versa).

5.5 Diversification

Our results show a clear trend towards varying and diversifying descriptions of intelligence (Figure 3): Each IUI conference published papers that referred to new entities as intelligent, accompanied by new co-descriptors. Such new references to intelligence might arise due to changing trend terms, new emerging technology and concepts, and more fine-grained specifications of previous ones (e.g. system vs. dialogue system). Overall, this matches the results of the analysis of Hornbæk et al. [33], who found diversifying characterisations for the concept of “interaction” in CHI proceedings. At the same time, we observed that most of the fundamental actions in our coding were present already early on. Together, these results thus paint the picture of a growing diversity of intelligent technology being used for fundamental actions in an interactive context, most prominently assisting and supporting users in their tasks.

5.6 Challenges

We collected nine challenges that emerged from the coded sentences. Together, they show the differentiated perspective of our community on intelligent technology: Technological intelligence is not presented as unconditionally beneficial but should rather be considerately integrated, weighing advantages and drawbacks with regard to user interaction. For example, although intelligent technology may help users to do certain tasks more efficiently, this support may come with usability issues and increase the overall complexity of interaction.

One of the identified challenges points to missing standards, both in terms of evaluation and interface design for intelligent technology. To date, the community often still seems to lack such standards and best practices, and future research might work towards finding ways to establish them.

5.7 Where Do We Go from Here?

Our work provides a survey of the status quo of the meaning of “intelligent” based on 25 years of active research in the IUI community. Interestingly, we found very few cases where researchers attempted to explicitly define how they understand intelligence in their work. Rather, our community currently assumes – or relies on – an implicit understanding of “intelligent”, which is also evident from the considerable number of IUI papers that never refer to “intelligen*” at all.

As one step towards more explicit engagement with the key terminology of our research subject, our analysis suggests that we can explicitly state how our assumptions and goals realise – or relate to – adaptation, automation, and interaction. These emerged as the three key aspects covered by our community’s understanding of “intelligent”. Moreover, explicitly stating, for example, where we position our work in the tools vs agents discussion can help us and the readers of our papers to contextualise our assumptions and goals around the involved intelligence.

That being said, we expect that insights and influences from HCI, AI, and related fields will continue to shape our understanding of “intelligent”, such as intelligence as the ability to learn (c.f. [14]). We therefore should continue the discussion of our understanding of the concept of intelligence and its key aspects beyond this paper.

6 Conclusion

More and more technology seems to be labelled as “intelligent” yet it has remained elusive what exactly renders an interactive system or user interface deserving of this term.

In this reflection paper, we analysed in detail the understanding of intelligence in the IUI community with its focus on research at the intersection of HCI and AI. In particular, we followed a bottom-up approach to analyse the emerging implicit perspective on characteristics of intelligence in intelligent user interfaces. To do so, we combined text scraping with manual analysis to examine 2011 sentences with “intelligen*” in the past 24 years of IUI proceedings.

Our analysis revealed that IUI researchers tend to describe intelligence in their work with the key aspects of automation, adaptation, and interaction. Moreover, researchers associate different descriptors with intelligent technology depending on how it is conceptualised (e.g. as a UI, system or agent).

These insights can help us as researchers at the intersection of HCI and AI to reflect on and more explicitly communicate our assumptions and goals around intelligence in our work. For example, we could explicitly label and describe aspects of adaptation and automation in our work, or make an explicit decision for presenting intelligence as part of a tool, UI, agent, assistant, and so on.

Overall, this paper thus for the first time provides a detailed reflection on the emerging collective understanding of intelligence as a key concept in this interdisciplinary community.

7 Acknowledgements

We thank ACM and the ACM Digital Library for providing us with full texts for all IUI proceedings. This project is funded by the Bavarian State Ministry of Science and the Arts in the framework of the Centre Digitisation.Bavaria (ZD.B).

References

  •  1. APA Definition of Intelligence. https://dictionary.apa.org/intelligence. Last access: 2019-10-01.
  •  2. A. Abdul, J. Vermeulen, D. Wang, B. Y. Lim, and M. Kankanhalli. Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 582:1–582:18, New York, NY, USA, 2018. ACM.
  •  3. R. S. Amant and P. R. Cohen. Interaction with a mixed-initiative system for exploratory data analysis. Knowledge-Based Systems, 10(5):265 – 273, 1998.
  •  4. K. Athukorala, A. Medlar, A. Oulasvirta, G. Jacucci, and D. Glowacka. Beyond relevance: Adapting exploration/exploitation in information retrieval. In Proceedings of the 21st International Conference on Intelligent User Interfaces, IUI ’16, pages 359–369, New York, NY, USA, 2016. ACM.
  •  5. D. Avrahami, M. Patel, Y. Yamaura, S. Kratz, and M. Cooper.

    Unobtrusive activity recognition and position estimation for work surfaces using rf-radar sensing.

    ACM Trans. Interact. Intell. Syst., 10(1), Aug. 2019.
  •  6. R. Bandelloni and F. Paternò. Flexible interface migration. In Proceedings of the 9th International Conference on Intelligent User Interfaces, IUI ’04, pages 148–155, New York, NY, USA, 2004. ACM.
  •  7. X. Bao, J. L. Herlocker, and T. G. Dietterich. Fewer clicks and less frustration: Reducing the cost of reaching the right folder. In Proceedings of the 11th International Conference on Intelligent User Interfaces, IUI ’06, pages 178–185, New York, NY, USA, 2006. ACM.
  •  8. L. Birnbaum, E. Horvitz, D. Kurlander, H. Lieberman, J. Marks, and S. Roth. Compelling intelligent user interfaces – how much ai? In Proceedings of the 2nd International Conference on Intelligent User Interfaces, IUI ’97, pages 173–175, New York, NY, USA, 1997. ACM.
  •  9. R. Bixler and S. D’Mello. Detecting boredom and engagement during writing with keystroke analysis, task appraisals, and stable traits. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, pages 225–234, New York, NY, USA, 2013. ACM.
  •  10. W. Bosma and E. André. Exploiting emotions to disambiguate dialogue acts. In Proceedings of the 9th International Conference on Intelligent User Interfaces, IUI ’04, pages 85–92, New York, NY, USA, 2004. ACM.
  •  11. S. Bourke, K. McCarthy, and B. Smyth. The social camera: A case-study in contextual image recommendation. In Proceedings of the 16th International Conference on Intelligent User Interfaces, IUI ’11, pages 13–22, New York, NY, USA, 2011. ACM.
  •  12. C. J. Cai, J. Jongejan, and J. Holbrook. The effects of example-based explanations in a machine learning interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19, pages 258–262, New York, NY, USA, 2019. ACM.
  •  13. P.-Y. Chi and H. Lieberman. Intelligent assistance for conversational storytelling using story patterns. In Proceedings of the 16th International Conference on Intelligent User Interfaces, IUI ’11, pages 217–226, New York, NY, USA, 2011. ACM.
  •  14. F. Chollet. On the measure of intelligence, 2019.
  •  15. R. Denaux, V. Dimitrova, L. Lau, P. Brna, D. Thakker, and C. Steiner. Employing linked data and dialogue for modelling cultural awareness of a user. In Proceedings of the 19th International Conference on Intelligent User Interfaces, IUI ’14, pages 241–246, New York, NY, USA, 2014. ACM.
  •  16. J. Dostal, P. O. Kristensson, and A. Quigley. Subtle gaze-dependent techniques for visualising display changes in multi-display environments. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, pages 137–148, New York, NY, USA, 2013. ACM.
  •  17. E. Edmonds. The future of intelligent interfaces: Not just “how?”, but “what?” and “why?”. In Proceedings of the 1st International Conference on Intelligent User Interfaces, IUI ’93, pages 13–17, New York, NY, USA, 1993. ACM.
  •  18. M. Eiband, H. Schneider, M. Bilandzic, J. Fazekas-Con, M. Haug, and H. Hussmann. Bringing transparency design into practice. In 23rd International Conference on Intelligent User Interfaces, IUI ’18, pages 211–223, New York, NY, USA, 2018. ACM.
  •  19. M. Eiband, S. T. Völkel, D. Buschek, S. Cook, and H. Hussmann. When people and algorithms meet: User-reported problems in intelligent everyday applications. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19, pages 96–106, New York, NY, USA, 2019. ACM.
  •  20. M. Eisenberg, A. Nishioka, and M. Schreiner. Helping users think in three dimensions: steps toward incorporating spatial cognition in user modelling. In Proceedings of the 2nd International Conference on Intelligent User Interfaces, volume 97 of IUI ’97, pages 113–120, New York, NY, USA, 1997. ACM.
  •  21. U. Farooq, J. Grudin, B. Shneiderman, P. Maes, and X. Ren. Human computer integration versus powerful tools. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’17, pages 1277–1282, New York, NY, USA, 2017. ACM.
  •  22. A. Faulring, B. Myers, K. Mohnkern, B. Schmerl, A. Steinfeld, J. Zimmerman, A. Smailagic, J. Hansen, and D. Siewiorek. Agent-assisted task management that reduces email overload. In Proceedings of the 15th International Conference on Intelligent User Interfaces, IUI ’10, pages 61–70, New York, NY, USA, 2010. ACM.
  •  23. J. E. Fischer, S. D. Ramchurn, M. Osborne, O. Parson, T. D. Huynh, M. Alam, N. Pantidi, S. Moran, K. Bachour, S. Reece, E. Costanza, T. Rodden, and N. R. Jennings. Recommending energy tariffs and load shifting based on smart household usage profiling. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, pages 383–394, New York, NY, USA, 2013. ACM.
  •  24. W. Fitzgerald, R. J. Firby, and M. Hannemann. Multimodal event parsing for intelligent user interfaces. In Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI ’03, pages 53–60, New York, NY, USA, 2003. ACM.
  •  25. K. Gajos and D. S. Weld. Supple: Automatically generating user interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces, IUI ’04, pages 93–100, New York, NY, USA, 2004. ACM.
  •  26. H. Gardner. Frames of mind: The theory of multiple intelligences. Basic Books, New York, NY, USA, 1983.
  •  27. H. Gardner. Multiple intelligences after twenty years, chapter 11, pages 87–94. Routledge, London, UK, 2006.
  •  28. A. Glass, D. L. McGuinness, and M. Wolverton. Toward establishing trust in adaptive agents. In Proceedings of the 13th International Conference on Intelligent User Interfaces, IUI ’08, pages 227–236, New York, NY, USA, 2008. ACM.
  •  29. D. Glowacka, T. Ruotsalo, K. Konuyshkova, K. Athukorala, S. Kaski, and G. Jacucci.

    Directing exploratory search: Reinforcement learning from user interactions with keywords.

    In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, pages 117–128, New York, NY, USA, 2013. ACM.
  •  30. F. M. Harper, D. Frankowski, S. Drenner, Y. Ren, S. Kiesler, L. Terveen, R. Kraut, and J. Riedl. Talk amongst yourselves: Inviting users to participate in online conversations. In Proceedings of the 12th International Conference on Intelligent User Interfaces, IUI ’07, pages 62–71, New York, NY, USA, 2007. ACM.
  •  31. K. Höök. Designing and evaluating intelligent user interfaces. In Proceedings of the 4th International Conference on Intelligent User Interfaces, IUI ’99, pages 5–6, New York, NY, USA, 1999. ACM.
  •  32. K. Höök. Steps to take before intelligent user interfaces become real. Interacting with Computers, 12(4):409 – 426, 2000.
  •  33. K. Hornbæk, A. Mottelson, J. Knibbe, and D. Vogel. What do we mean by “interaction”? an analysis of 35 years of chi. ACM Trans. Comput.-Hum. Interact., 26(4), July 2019.
  •  34. K. Hornbæk and A. Oulasvirta. What is interaction? In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, pages 5040–5052, New York, NY, USA, 2017. ACM.
  •  35. D. A. Joyner, A. K. Goel, and N. M. Papin. Mila–s: Generation of agent-based simulations from conceptual models of complex systems. In Proceedings of the 19th International Conference on Intelligent User Interfaces, IUI ’14, pages 289–298, New York, NY, USA, 2014. ACM.
  •  36. K. Katsuragawa, A. Kamal, and E. Lank. Effect of motion-gesture recognizer error pattern on user workload and behavior. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI ’17, pages 439–449, New York, NY, USA, 2017. ACM.
  •  37. V. Kostakos. The big hole in hci research. interactions, 22(2):48–51, Feb. 2015.
  •  38. K. Kotowick and J. Shah. Intelligent sensory modality selection for electronic supportive devices. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI ’17, pages 55–66, New York, NY, USA, 2017. ACM.
  •  39. P. O. Kristensson and S. Zhai.

    Relaxing stylus typing precision by geometric pattern matching.

    In Proceedings of the 10th International Conference on Intelligent User Interfaces, IUI ’05, pages 151–158, New York, NY, USA, 2005. ACM.
  •  40. P. Lanz. The concept of intelligence in psychology and philosophy. In H. Cruse, J. Dean, and H. Ritter, editors, Prerational Intelligence: Adaptive Behavior and Intelligent Systems Without Symbols and Logic, Volume 1, Volume 2 Prerational Intelligence: Interdisciplinary Perspectives on the Behavior of Natural and Artificial Systems, Volume 3, pages 19–30. Springer, Dordrecht, Netherlands, 2000.
  •  41. C. Lee, A. Garbett, J. Wang, B. Hu, and D. Jackson. Weaving the topics of chi: Using citation network analysis to explore emerging trends. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, pages 1–6, New York, NY, USA, 2019. ACM.
  •  42. S. Lefort, E. Zibetti, M.-J. Lesot, M. Detyniecki, and C. Tijus. Dimensions for automatic interpretation of approximate numerical expressions: An empirical study. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI ’17, pages 107–117, New York, NY, USA, 2017. ACM.
  •  43. J. Li, M. X. Zhou, H. Yang, and G. Mark. Confiding in and listening to virtual agents: The effect of personality. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI ’17, pages 275–286, New York, NY, USA, 2017. ACM.
  •  44. Y. Liu, J. Goncalves, D. Ferreira, B. Xiao, S. Hosio, and V. Kostakos. Chi 1994–2013: Mapping two decades of intellectual progress through co-word analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, pages 3553–3562, New York, NY, USA, 2014. ACM.
  •  45. S. Lok and S. K. Feiner. The ail automated interface layout system. In Proceedings of the 7th International Conference on Intelligent User Interfaces, IUI ’02, pages 202–203, New York, NY, USA, 2002. ACM.
  •  46. D. MacLean, S. Hangal, S. K. Teh, M. S. Lam, and J. Heer. Groups without tears: Mining social topologies from email. In Proceedings of the 16th International Conference on Intelligent User Interfaces, IUI ’11, pages 83–92, New York, NY, USA, 2011. ACM.
  •  47. N. Mauro and L. Ardissono. Session-based suggestion of topics for geographic exploratory search. In 23rd International Conference on Intelligent User Interfaces, IUI ’18, pages 341–352, New York, NY, USA, 2018. ACM.
  •  48. C. A. Miller. Computational approaches to interface design: What works, what doesn’t, what should and what might. In Proceedings of the 2nd International Conference on Intelligent User Interfaces, IUI ’97, pages 123–126, New York, NY, USA, 1997. ACM.
  •  49. E. Mynatt and J. Tullio. Inferring calendar event attendance. In Proceedings of the 6th International Conference on Intelligent User Interfaces, IUI ’01, pages 121–128, New York, NY, USA, 2001. ACM.
  •  50. D. Nagel. The opportunity of a new century. In Proceedings of the 3rd International Conference on Intelligent User Interfaces, IUI ’98, page 9, New York, NY, USA, 1998. ACM.
  •  51. D. Norman. Complexity versus difficulty: Where should the intelligence be? In Proceedings of the 7th International Conference on Intelligent User Interfaces, IUI ’02, page 4, New York, NY, USA, 2002. ACM.
  •  52. E. Ohn-Bar, J. a. Guerreiro, D. Ahmetovic, K. M. Kitani, and C. Asakawa. Modeling expertise in assistive navigation interfaces for blind people. In 23rd International Conference on Intelligent User Interfaces, IUI ’18, pages 403–407, New York, NY, USA, 2018. ACM.
  •  53. R. Ovans and W. S. Havens. Intelligent mediation: An architecture for the real-time allocation of interface resources. In Proceedings of the 1st International Conference on Intelligent User Interfaces, IUI ’93, pages 55–61, New York, NY, USA, 1993. ACM.
  •  54. S. Penney, J. Dodge, C. Hilderbrand, A. Anderson, L. Simpson, and M. Burnett. Toward foraging for understanding of starcraft agents: An empirical study. In 23rd International Conference on Intelligent User Interfaces, IUI ’18, pages 225–237, New York, NY, USA, 2018. ACM.
  •  55. A. R. Puerta. The study of models of intelligent interfaces. In Proceedings of the 1st International Conference on Intelligent User Interfaces, IUI ’93, pages 71–78, New York, NY, USA, 1993. ACM.
  •  56. L. Qu, N. Wang, and W. L. Johnson. Choosing when to interact with learners. In Proceedings of the 9th International Conference on Intelligent User Interfaces, IUI ’04, pages 307–309, New York, NY, USA, 2004. ACM.
  •  57. C. Rich and C. L. Sidner. Segmented interaction history in a collaborative interface agent. In Proceedings of the 2nd International Conference on Intelligent User Interfaces, IUI ’97, pages 23–30, New York, NY, USA, 1997. ACM.
  •  58. J. Shen, E. Fitzhenry, and T. G. Dietterich. Discovering frequent work procedures from resource connections. In Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI ’09, pages 277–286, New York, NY, USA, 2009. ACM.
  •  59. B. Shneiderman and P. Maes. Direct manipulation vs. interface agents. Interactions, 4(6):42–61, Nov. 1997.
  •  60. M. P. Singh. Multiagent Systems, pages 1–14. Springer, Berlin, Heidelberg, Germany, 1994.
  •  61. D. A. Smith and H. Lieberman. The why ui: Using goal networks to improve user interfaces. In Proceedings of the 15th International Conference on Intelligent User Interfaces, IUI ’10, pages 377–380, New York, NY, USA, 2010. ACM.
  •  62. R. R. Sokal. A statistical method for evaluating systematic relationships. University of Kansas Science Bulletin, 38:1409–1438, 1958.
  •  63. A. Springer and S. Whittaker. Progressive disclosure: Empirically motivated approaches to designing effective transparency. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19, pages 107–120, New York, NY, USA, 2019. ACM.
  •  64. C. Stephanidis, C. Karagiannidis, and A. Koumpis. Decision making in intelligent user interfaces. In Proceedings of the 2nd International Conference on Intelligent User Interfaces, IUI ’97, pages 195–202, New York, NY, USA, 1997. ACM.
  •  65. R. J. Sternberg. Human intelligence. Encyclopaedia Britannica; https://www.britannica.com/science/human-intelligence-psychology. Last access: 2019-10-01.
  •  66. R. J. Sternberg. The concept of intelligence and its role in lifelong learning and success. American Psychologist, 52(10):1030–1037, 1997.
  •  67. M. Sun and J. Y. Chai. Towards intelligent qa interfaces: Discourse processing for context questions. In Proceedings of the 11th International Conference on Intelligent User Interfaces, IUI ’06, pages 163–170, New York, NY, USA, 2006. ACM.
  •  68. P. Szekely. Structuring Programs to Support Intelligent Interfaces, pages 445–464. ACM, New York, NY, USA, 1991.
  •  69. C. S. S. Tan, J. Schöning, K. Luyten, and K. Coninx. Informing intelligent user interfaces by inferring affective states from body postures in ubiquitous computing environments. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, pages 235–246, New York, NY, USA, 2013. ACM.
  •  70. M. I. Tanveer, E. Lin, and M. E. Hoque. Rhema: A real-time in-situ intelligent interface to help people with public speaking. In Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI ’15, pages 286–295, New York, NY, USA, 2015. ACM.
  •  71. M. I. Tanveer, R. Zhao, K. Chen, Z. Tiet, and M. E. Hoque. Automanner: An automated interface for making public speakers aware of their mannerisms. In Proceedings of the 21st International Conference on Intelligent User Interfaces, IUI ’16, pages 385–396, New York, NY, USA, 2016. ACM.
  •  72. A. M. Turing. Computing machinery and intelligence. Mind, 59(236):433–433, 1950.
  •  73. K. Vertanen and P. O. Kristensson. Parakeet: A continuous speech recognition system for mobile touch-screen devices. In Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI ’09, pages 237–246, New York, NY, USA, 2009. ACM.
  •  74. W. Wahlster and M. Maybury. An Introduction to Intelligent User Interfaces. In RUIU, pages 1–13. Morgan Kaufmann, San Francisco, CA, USA, 1998.
  •  75. W.-K. Wong, I. Oberst, S. Das, T. Moore, S. Stumpf, K. McIntosh, and M. Burnett. End-user feature labeling: A locally-weighted regression approach. In Proceedings of the 16th International Conference on Intelligent User Interfaces, IUI ’11, pages 115–124, New York, NY, USA, 2011. ACM.
  •  76. D. D. Woods. The price of flexibility. In Proceedings of the 1st International Conference on Intelligent User Interfaces, IUI ’93, pages 19–25, New York, NY, USA, 1993. ACM.
  •  77. Z. Xiao, M. X. Zhou, and W.-T. Fu.

    Who should be my teammates: Using a conversational agent to understand individuals and help teaming.

    In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19, pages 437–447, New York, NY, USA, 2019. ACM.
  •  78. M. X. Zhou and K. Houck. A semantic approach to the dynamic design of interaction controls in conversation systems. In Proceedings of the 7th International Conference on Intelligent User Interfaces, IUI ’02, pages 167–174, New York, NY, USA, 2002. ACM.