Human's Role in-the-Loop

04/27/2022
by   Avigdor Gal, et al.
Technion
0

Data integration has been recently challenged by the need to handle large volumes of data, arriving at high velocity from a variety of sources, which demonstrate varying levels of veracity. This challenging setting, often referred to as big data, renders many of the existing techniques, especially those that are human-intensive, obsolete. Big data also produces technological advancements such as Internet of things, cloud computing, and deep learning, and accordingly, provides a new, exciting, and challenging research agenda. Given the availability of data and the improvement of machine learning techniques, this blog discusses the respective roles of humans and machines in achieving cognitive tasks in matching, aiming to determine whether traditional roles of humans and machines are subject to change. Such investigation, we believe, will pave a way to better utilize both human and machine resources in new and innovative manners. We shall discuss two possible modes of change, namely humans out and humans in. Humans out aim at exploring out-of-the-box latent matching reasoning using machine learning algorithms when attempting to overpower human matcher performance. Pursuing out-of-the-box thinking, machine and deep learning can be involved in matching. Humans in explores how to better involve humans in the matching loop by assigning human matchers with a symmetric role to algorithmic matcher in the matching process.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

page 3

page 4

11/30/2020

Human vs. supervised machine learning: Who learns patterns faster?

The capabilities of supervised machine learning (SML), especially compar...
07/25/2019

HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop

While the role of humans is increasingly recognized in machine learning ...
08/27/2020

Computational Models of Human Decision-Making with Application to the Internet of Everything

The concept of the Internet of Things (IoT) first appeared a few decades...
12/02/2020

Learning to Characterize Matching Experts

Matching is a task at the heart of any data integration process, aimed a...
08/26/2020

Ethical behavior in humans and machines – Evaluating training data quality for beneficial machine learning

Machine behavior that is based on learning algorithms can be significant...
08/03/2017

A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop

The goal of Machine Learning to automatically learn from data, extract k...
04/07/2020

Neural Analogical Matching

Analogy is core to human cognition. It allows us to solve problems based...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

The research of data integration spans over multiple decades, holds both theoretical and practical appeal, and enjoys a continued interest by researchers and practitioners. Efforts in the area include, among other things, process matching, schema matching, and entity resolution. Matching problems have been historically treated as semi-automated tasks in which correspondences are generated by automatic algorithms and subsequently validated by human expert(s). The reason for that is the inherent assumption that humans “do it better.”

Data integration has been recently challenged by the need to handle large volumes of data, arriving at high velocity from a variety of sources, which demonstrate varying levels of veracity. This challenging setting, often referred to as big data, renders many of the existing techniques, especially those that are human-intensive, obsolete. Big data also produces technological advancements such as Internet of things, cloud computing, and deep learning, and accordingly, provides a new, exciting, and challenging research agenda.

Given the availability of data and the improvement of machine learning techniques, this blog discusses the respective roles of humans and machines in achieving cognitive tasks in matching, aiming to determine whether traditional roles of humans and machines are subject to change  (Shraga, 2018, 2020). Such investigation, we believe, will pave a way to better utilize both human and machine resources in new and innovative manners.

We shall discuss two possible modes of change, namely humans out and humans in. Humans out aim at exploring out-of-the-box latent matching reasoning using machine learning algorithms when attempting to overpower human matcher performance. Pursuing out-of-the-box thinking, machine and deep learning can be involved in matching. Humans in explores how to better involve humans in the matching loop by assigning human matchers with a symmetric role to algorithmic matcher in the matching process.

We start by scoping the discussion and introduce the matching problem. Then, we separate the discussion to the role of humans in-the-loop to humans out and humans in.

2. Matching 101

Modern industrial and business processes require intensive use of large-scale data integration techniques to combine data from multiple heterogeneous data sources into meaningful and valuable information. Such integration is performed on structured and semi-structured datasets from various sources such as SQL and XML schemata, entity-relationship diagrams, ontology descriptions, Web service specifications, interface definitions, process models, and Web forms. Data integration plays a key role in a variety of domains, including data warehouse loading and exchange, data wrangling, data lakes, aligning ontologies for the Semantic Web, Web service composition, and business document format merging (e.g., orders and invoices in e-commence).

PO

City

poCode

city

DateTime

poDay

poTime

poCode

PO

Order_Details

orderNumber

city

price

orderDate
Figure 1. Schema Matching Example.

A major challenge in data integration is a matching task, which creates correspondences between model elements, may they be schema attributes (see, for example, Figure 1

), ontology concepts, model entities, or process activities. Matching research has been a focus for multiple disciplines including Databases, Artificial Intelligence, Semantic Web, Process Management, and Data Mining. Most studies have focused on designing high quality matchers, automatic tools for identifying correspondences. Several heuristic attempts were followed by theoretical grounding, showing that matching is inherently an uncertain decision making process due to ambiguity and heterogeneity of structure, semantics, and forms of representation of identical concepts.

Figure 2. Similarity Matrix Adjustment Illustration.

We position the matching task as a similarity matrix adjustment process (see Figure 2 for an illustration). A matcher’s output is conceptualized as a (possibly high dimensional) similarity matrix, in which a matcher records the similarity of elements from multiple data sources. Such elements can be attributes in a relational schema, tuples in databases that possibly represent the same real-world entity, etc. A major benefit of this approach is the ability to view schema matching as a machine learning task, paving the way to utilize deep learning models and achieve significant improvements in the ability to correctly match real-world data.

The quality of automatic matching outcome is typically assessed using some evaluation metric (

e.g., Precision, Recall, F1). Applying such metrics usually requires human involvement to validate the decisions made by automatic matchers. (Zhang et al., 2018). Yet, human validation of matching requires domain expertise (Dragisic et al., 2016) and may be laborious, (Zhang et al., 2018), biased, (Ackerman et al., 2019), and diverse. (Ross et al., 2010; Shraga et al., 2018)

. This, in turn, limits the amount of qualitative labels that can be provided for supervised learning, especially when new domains are introduced. Matching predictors have been proposed in the literature as alternative evaluators for matching outcome, opting to correlate well with evaluation metrics created from human judgment.

It becomes clear that while matching tasks have been largely dependent on human validation, human performance in matching and validation does not meet the expectations of those who would like matches with high confidence. For many years there was no other option rather than trusting humans and secretly hoping our experts indeed perform well. The introduction of effective machine longer offers a promise of a better matching world, as will be discussed next.

3. Humans Out

The Humans Out

approach seeks matching subtasks, traditionally considered to require cognitive effort, in which humans can be excluded. An initial good place to start is with the basic task of identifying correspondences. We note that many contemporary matching algorithms use heuristics, where each heuristic applies some semantic cue to justify an alignment between elements. For example, string-based matchers use string similarity as a cue for item alignment. We observe that such heuristics, in essence, encode human intuition about matching. Research shows that human matching choices can be reasonably predicted by classifying them into types, where a type corresponds to an existing heuristic. Therefore, we can argue that some of the cognitive effort of human matchers can be replaced by automated solutions. We next briefly describe two examples aiming to enhance the automation of matching using machine learning.

Learning to Rerank Matches: Choosing the best match from a top- is basically a cognitive task, usually done by humans. A recent work offers an algorithmic replacement to humans in selecting the best match (Gal et al., 2019, 2018), a task traditionally reserved for human verifiers. The novelty of this work is in the use of similarity matrices as a basis for learning features, creating feature-rich datasets that fit learning and enriches algorithmic matching beyond that of human matching. The suggested ranking framework adopts a learning-to-rank approach, utilizing matching predictors as features. An interesting aspect of this work is a bound on the size of , given a desired level of confidence in finding the best match, justified theoretically and validated empirically. This bound is useful for top- algorithms and, as psychological literature suggests (Schwartz and Schwartz, 2004), also applicable when introducing a list of options to humans.

Figure 3. Deep Similarity Matrix Adjustment and Evaluation Example.

Deep Similarity Matrix Adjustment and Evaluation: Traditionally, the adjustment and evaluation of matching results were done heuristically or by humans. A recent paper shows that these two processes can benefit from deep learning techniques (Shraga et al., 2020a)

. The work offers a novel post processing step for schema matching that manipulates, using deep neural networks, similarity matrices, created by state-of-the-art algorithmic matchers (see Figure 

3 for example). The suggested methodology provides a data-driven approach for extracting hidden representative features for an automatic schema matching process, removing the requirement for manual feature engineering. Moreover, it enhances the ability to introduce new data sources to existing systems without the need to rely on either domain experts (knowledgeable of the domain but less so on the best matchers to use) or data integration specialists (who lack sufficient domain knowledge).

4. Humans In

The Humans In approach aims at investigating whether the current role humans take in the matching process is effective (spoiler: it is not) and whether an alternative role can improve overall performance of the matching process (spoiler: it can). A recent study, aided by metacognitive models, analyzes the consistency of human matchers (Ackerman et al., 2019). Three main consistency dimensions were explored as potential cognitive biases, taking into account the time it takes to reach a matching decision, the extent of agreement among human matchers and the assistance of algorithmic matchers. In particular, it was shown that when an algorithmic suggestion is available, humans tend to accept it to be true, in sharp contradiction to the conventional validation role of human matchers.

Figure 4. Incognitomatch User Statistics.

All these dimensions were found predictive of both confidence and accuracy of human matchers. Specifically, these biases can be used to recognize and capture predictive behaviors in the human decision making process (e.g., decision times and screen scrolls by mouse tracking). A recent research provides a machine learning framework to characterize domain experts that can be used to identify reliable and valuable human experts, those humans whose proposed correspondences can mostly be trusted to be valid (Shraga et al., 2021; Shraga and Gal, 2021).

A new matching crowdsourcing application (see Figure 4 for an example screenshot), called Incognitomatch (Shraga et al., 2020b)111https://agp.iem.technion.ac.il/incognitomatch/, was launched to support the gathering of behavioral information and offer support in analyzing human matching behavior.

5. Conclusion

In this blog we presented our approach for human involvement in the matching loop, introducing tasks where humans can be replaced and emphasizing our vision for understanding human behavior to allow better engagement. An additional overarching goal is to propose a common matching framework that would allow treating matching as a unified problem whether we match schemata attributes, ontology elements, process activities, entity’s tuples, etc.

We envision future research to rely on machine and deep learning to generate new and innovative methods to improve matching problems, making extensive use of similarity matrices. Using machine learning for this purpose immediately raises the issue of shortage of labeled data to offer supervised learning. Hence, pursuing less-than-supervised (e.g., unsupervised, weakly supervised) methods would be a natural next step to pursue.

About the Authors

Avigdor Gal is a Professor of Data Science at the Technion – Israel Institute of Technology. He is also the Academic Director of Data Science & Engineering for the university’s undergraduate and graduate programs, a member of the management team of the Technion’s Data Science Initiative and the head of the Big Data Integration laboratory in the Technion. In the current age of big data, his research is focused on developing novel models and algorithms for data integration broadly construed, and in particular the investigation of aspects of uncertainty when integrating data from multiple data sources. He is an expert in information systems and data science, and more recently, in machine learning and artificial intelligence.

Roee Shraga is a Postdoctoral fellow at the Technion – Israel Institute of Technology, from which he received a PhD degree in 2020 in the area of Data Science. Roee has published more than a dozen papers in leading journals and conferences on the topics of data integration, human-in-the-loop, machine learning, process mining, and information retrieval. He is also a recipient of several PhD fellowships including the Leonard and Diane Sherman Interdisciplinary Fellowship (2017), the Daniel Excellence Scholarship (2019), and the Miriam and Aaron Gutwirth Memorial Fellowship (2020).

Acknowledgments

We would like thank Prof. Rakefet Ackerman, Dr. Haggai Roitman, Dr. Tomer Sagi, Dr. Ofra Amir,and Coral Scharf for their involvement in our research.

References

  • R. Ackerman, A. Gal, T. Sagi, and R. Shraga (2019) A cognitive model of human bias in matching. In Pacific Rim International Conference on Artificial Intelligence, pp. 632–646. Cited by: §2, §4.
  • Z. Dragisic, V. Ivanova, P. Lambrix, D. Faria, E. Jiménez-Ruiz, and C. Pesquita (2016) User validation in ontology alignment. In International Semantic Web Conference, pp. 200–217. Cited by: §2.
  • A. Gal, H. Roitman, and R. Shraga (2018) Heterogeneous data integration by learning to rerank schema matches. In Proceedings of the International Conference on Data Mining, ICDM, pp. 959–964. Cited by: §3.
  • A. Gal, H. Roitman, and R. Shraga (2019) Learning to rerank schema matches. IEEE Transactions on Knowledge and Data Engineering, TKDE. Cited by: §3.
  • J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson (2010) Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI’10 Extended Abstracts on Human Factors in Computing Systems, pp. 2863–2872. Cited by: §2.
  • B. Schwartz and B. Schwartz (2004) The paradox of choice: why more is less. Cited by: §3.
  • R. Shraga, O. Amir, and A. Gal (2021) Learning to characterize matching experts. In Proceedings of the International Conference on Data Engineering, ICDE, Cited by: §4.
  • R. Shraga, A. Gal, and H. Roitman (2018) What type of a matcher are you?: coordination of human and algorithmic matchers. See DBLP:conf/sigmod/2018hilda, pp. 12:1–12:7. Cited by: §2.
  • R. Shraga, A. Gal, and H. Roitman (2020a) ADnEV: cross-domain schema matching using deep similarity matrix adjustment and evaluation. Proceedings of the VLDB Endowment 13 (9), pp. 1401–1415. Cited by: §3.
  • R. Shraga and A. Gal (2021) PoWareMatch: a quality-aware deep learning approach to improve human schema matching. Journal of Data and Information Quality, JDIQ. Cited by: §4.
  • R. Shraga, C. Scharf, R. Ackerman, and A. Gal (2020b) InCognitoMatch: cognitive-aware matching via crowdsourcing. In Proceedings of the International Conference on Management of Data, SIGMOD (36% acceptance rate), pp. 2753––2756. Cited by: §4.
  • R. Shraga (2018) (Artificial) mind over matter: integrating humans and algorithms in solving matching problems. In Proceedings of the International Conference on Management of Data, SIGMOD, pp. 1813–1815. Cited by: §1.
  • R. Shraga (2020) (Artificial) mind over matter: humans in and humans out in matching. In Proceedings of the VLDB 2020 PhD Workshop, Cited by: §1.
  • C. Zhang, L. Chen, H. Jagadish, M. Zhang, and Y. Tong (2018) Reducing uncertainty of schema matching via crowdsourcing with accuracy rates. IEEE Transactions on Knowledge and Data Engineering. Cited by: §2.