DeepAI
Log In Sign Up

Towards a Reference Software Architecture for Human-AI Teaming in Smart Manufacturing

With the proliferation of AI-enabled software systems in smart manufacturing, the role of such systems moves away from a reactive to a proactive role that provides context-specific support to manufacturing operators. In the frame of the EU funded Teaming.AI project, we identified the monitoring of teaming aspects in human-AI collaboration, the runtime monitoring and validation of ethical policies, and the support for experimentation with data and machine learning algorithms as the most relevant challenges for human-AI teaming in smart manufacturing. Based on these challenges, we developed a reference software architecture based on knowledge graphs, tracking and scene analysis, and components for relational machine learning with a particular focus on its scalability. Our approach uses knowledge graphs to capture product- and process specific knowledge in the manufacturing process and to utilize it for relational machine learning. This allows for context-specific recommendations for actions in the manufacturing process for the optimization of product quality and the prevention of physical harm. The empirical validation of this software architecture will be conducted in cooperation with three large-scale companies in the automotive, energy systems, and precision machining domain. In this paper we discuss the identified challenges for such a reference software architecture, present its preliminary status, and sketch our further research vision in this project.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

05/31/2022

Quality Characteristics of a Software Platform for Human-AI Teaming in Smart Manufacturing

As AI-enabled software systems become more prevalent in smart manufactur...
08/07/2021

Artificial Intelligence-Driven Customized Manufacturing Factory: Key Technologies, Applications, and Challenges

The traditional production paradigm of large batch production does not o...
05/12/2020

Computer Vision Toolkit for Non-invasive Monitoring of Factory Floor Artifacts

Digitization has led to smart, connected technologies be an integral par...
03/02/2015

A Review of Relational Machine Learning for Knowledge Graphs

Relational machine learning studies methods for the statistical analysis...
05/04/2021

The Role of Time and Data: Online Conformance Checking in the Manufacturing Domain

Process mining has matured as analysis instrument for process-oriented d...
10/26/2022

Scaling Knowledge Graphs for Automating AI of Digital Twins

Digital Twins are digital representations of systems in the Internet of ...
04/19/2020

Machine Learning based Pallets Detection and Tracking in AGVs

The use of automated guided vehicles (AGVs) has played a pivotal role in...

1. Introduction

Applications of AI in smart manufacturing are manifold, ranging from improving maintenance times of machinery, the detection of failures in the product or the machinery to the prevention of harm to manufacturing operators. In general, complex processes that are worked on collaboratively are characterized by a sequence of reactive and proactive elements, which each actor alternatingly supporting the other. AI-enabled systems in smart manufacturing are capable of self-sensing, self-adaptation, self-organization, and self-decision (Qu et al., 2019; Phuyal et al., 2020), allowing them to respond to physical changes in the production environment in various ways - by stopping machines, adapting production tasks, or suggesting the change of production parameters. However, effective teaming between manufacturing operators and AI-enabled manufacturing systems requires mutual trust into the other’s capabilities, primarily resulting from self-sensing and self-adaption. With regards to collaborative AI systems, this demands a high degree of situational awareness for each other actor’s needs, knowledge of the production process, and its adjustable parameters.

From a higher perspective, two main challenges related to teaming AI and manufacturing operators in smart manufacturing can be identified: The first challenge relates to the required scalability of the architecture when processing data in near-realtime, particulary in combination with relational machine learning, i.e., the statistical analysis of relational, or graph-structured, data (Nickel et al., 2016). The second challenge relates to examining a suitable framework to explicate the knowledge for effective teaming in the manufacturing process. Shared mental models capture the common ground knowledge in the collaboration between humans with robots (Scheutz et al., 2017; Gervits et al., 2018; Jonker et al., 2011). We use knowledge graphs and ontologies to formalize these shared mental models of the manufacturing process and the semantics of trust factors for human-AI teaming in an operational manner.

In this paper we first present the initial reference software architecture as one important contribution of the Teaming.AI111https://www.teamingai-project.eu project. The remainder of the paper is structured as follows: First we sketch the envisioned approach based on a use case from one of the industrial partners in Section 2, before we elaborate the identified challenges for such a reference architecture in Section 3. Afterwards, we present the current status of this software architecture in Section 4. Finally, we sketch our next research activities and conclude the paper in Section 5.

2. Envisioned Approach

One of our industrial partners specializes in high-precision machining of large-sized parts by milling or grinding based on cast materials or machine-welded structures. Manufacturing operators must manually clamp large and heavy parts into high-precision manufacturing machines to perform grinding or milling operations. This process takes up a large portion of the total cycle time of a work order and workers are exposed to occupational hazards. A reliable full automation by an AI-assisted robotic system is beyond the capabilities of current AI-based technologies, due to large part variety and the complexity of the associated handling processes.

By compensating for the limited visual perception of both actors in the manufacturing process - the manufacturing operator and the AI system - we seek to maximize awareness of hazardous situations and conditions that negatively impact physical or mental ergonomics. Based on human feedback, the AI system learns to predict which action sequences are ergonomically favorable. However, since the AI-augmented manufacturing system cannot be sure that it has the complete information about the situation due to occlusions and unseen areas, it expresses its confidence to the manufacturing operator by, for example, displaying safety warnings. The system fuses observational data from a visual tracking system with prior knowledge about the scene and process with in-process feedback from the manufacturing operator. Obviously, for such a system to gain the trust from the manufacturing operator, not only the correctness, and context-dependent suitability are critical success factors, but also the timely provision of recommended actions or the display of safety warnings. Of course, such a system must also comply with ethical policies and standards, which the manufacturing operator also tacitly assumes.

The general approach envisioned in our research project uses these data for machine learning and the development of a context-aware AI-augmented manufacturing system. Thereby, prior knowledge about the scene, geometry of the parts, and compositional workflow patterns for the handling and processing steps are encoded in a knowledge graph, along with safety guidelines. The knowledge graph is updated at runtime with current contextual information. An integrated self-diagnostic component simultaneously assesses the system’s level of consistency and completeness and expresses its epistemic confidence. In case of ambiguity, the manufacturing operator is asked for feedback.

From a functional perspective, the teaming of manufacturing operators and AI in an industrial manufacturing process raises specific challenges related to relational machine learning, processing IIoT streaming data, the modeling and monitoring of trust (factors), and the continuous evaluation of human feedback and compliance with ethical policies. From a non-functional perspective, scalability and timing requirements of the components play a special role for meeting the functional requirements in such a system.

3. Research Challenges

In multiple workshops over the course of six months our research consortium identified five core challenges relevant for a reference architecture for human-AI teaming in smart manufacturing. These workshops involved researchers from software engineering (3), knowledge engineering (3), machine learning (4), computer vision (2), vision systems (2), and human factors (1) and domain experts from the companies, such as production and quality managers.


1
Monitoring of Teaming Aspects in Human-AI Interaction. Salsas et al. in (Salas et al., 2005) identified five core components that are necessary for effective teamwork. In addition, shared mental models (Jonker et al., 2011), mutual trust, and closed-loop communication serve as coordination mechanism between the actors. We use these concepts as foundation for modeling the manufacturing process as alternating activities between the manufacturing operator and the AI-augmented manufacturing system. Further, we rely on the 4S Interdependence Framework to define the neccessary state, structural organization, skills, and strategy for each actor and activitiy in the smart manufacturing process. The monitoring of these aspects, e.g., through mining event logs or scene detection through a tracking and scene analysis system, is the main challenge in this context.

2
Scalability for Near-Realtime IIoT Data Processing. In smart manufacturing, streaming data acquired from IIoT sensors continuously need to be processed so that they can be used for machine learning. Thus, the scalability of a reference architecture (Khalil et al., 2021; Santos et al., 2019) remains a key challenge in this context as delayed or incorrect recommendations of the AI systems to manufacturing operators may delay the workflow or, even worse, can also endanger their safety. In this context we research on appropriate consistency requirements in the face of data sharding between different replicas to better cope with typical data volumes in this context.

3
Runtime Monitoring and Validation of Ethical Policies Formalizing ethical and standards in an operational manner and validating their fulfillment by an AI system at runtime remains a key challenge (Vakkuri et al., 2019, 2021; Hagendorff, 2020; Johnson and Smith, 2021) in this context. The continuous validation of fulfillment of these policies and standards is exacerbated by the fact that they can only be evaluated at runtime. In our research we aim on developing a formalism based on ontology-reasoning to express these policies and standards in an operational manner.

4
Relational Machine Learning for Knowledge Graphs. The first research challenge relates to examining different approaches for calculating graph embeddings and their respective computational costs (Nickel et al., 2016; Trouillon et al., 2017). Second, for the development of the framework abstraction layer, the required functionalities for orchestrated machine learning need to be identified, which is the main challenge in this context. Subsequently, generic interfaces covering these common functionalities can be derived thereof.

5
Experimentation of Data and Algorithms. The automatization of experiments requires an operational definition of the experiments’ evaluation criteria, the parameterization of required data pipelines and machine learning algorithms, and the configuration of their deployment and execution. Thus, the research challenge comprises the development of domain-specific languages to describe and evaluate these experiments based on quality factors of AI-based systems (Siebert et al., 2020) in an operational manner. Further it comprises the development of respective tooling for automatizing the evaluation of the experiments (Nguyen-Duc and Abrahamsson, 2020; Bosch et al., 2020) that also integrates qualitative expert feedback from the human-in-the-loop (HITL).

4. Preliminary Status of the Reference Software Architecture

Figure 1. Reference Software Architecture For Human-AI Teaming in Smart Manufacturing.

Reference Software Architecture For Human-AI Teaming in Smart Manufacturing.

Based on the aforementioned challenges our research consortium developed a reference software architecture that serves as a blueprint for our subsequent research activities and validations. Though this architecture merges different viewpoints from researchers with software engineering and machine learning backgrounds, we expect subtle changes with progress of the research project. Thus, this description captures its preliminary status.

Figure 1 shows the different components of this reference architecture. To account for the different latency requirements of the components to process the data in a streaming-like manner, we followed the Lambda architecture pattern as described by Warren and Marz (Warren and Marz, 2015). This architectural pattern groups the components based on their latency requirements into three layers. The batch layer (model authoring) ingests and stores large amounts of data, the speed layer (knowledge graph, graph-based ML and teaming engine, production line systems) processes updates to the data in low-latency, and the serving layer (operation support, ML experimentation, introspection & policy monitoring) provides precalculated results also in a low-latency fashion (Preuveneers et al., 2016). To separate read and write operations and therewith be able to balance the processing of large data volumes, all data stores used in the architecture (i.e., dynamic knowledge graph, time series, and media data) are replicated as read and write shards. The synchronization between these replicas is performed autonomously by the synchronization management component.

4.1. Model Authoring & Import

Product- and process specific models for the manufacturing process are imported into the knowledge graph by means of batches. In this regard, importing might also comprise to convert these domain-specific models into a graph representation such as RDF (Resource Description Framework). This component also embodies the functionality needed for authoring and versioning the models.

4.2. ML Experimentation

During the continuous improvement and testing of AI-based systems, huge amounts of data are generated that can be used for targeted experiments with these systems (Arpteg et al., 2018; Amershi et al., 2019). Particularly, the indeterministic nature of data-driven algorithms and their entanglement with the used training data underpins the need for continuous experimentation and validation of AI components.

4.3. Introspection & Policy Monitoring

Explainability of AI (XAI) (Rudin, 2019), ethics-based auditability (Mökander et al., 2021), and the alignment with human rights and autonomy (IEEE, 2019) are common examples for ethical requirements towards AI systems. In recent years, several policies and guidelines have been presented by companies, e.g., Google (Pichai, 2018), governmental bodies, e.g., the EU (HLEG, 2019), and standardization organizations, e.g., from IEEE (IEEE, 2019). The compliance with these standards and policies needs to be assured throughout the operation of any AI-enabled software system. To this end, this component encapsulates introspection capabilities, i.e., the self-directed evaluation of the AI system by itself, and additional monitoring and validation facilities related to ethical standards and policies. While introspection is triggered ad-hoc, e.g., upon the detection of suspicious interaction patterns, monitoring and validation is performed continuously at runtime. To also take into account historical data when evaluating the compliance with ethical policies, all processed events are stored in the event store.

4.4. Knowledge Graph

Due to the emerging use of knowledge graphs to represent ontologies and semantic data, they also become increasingly important to formalize expert knowledge, process, and simulation data in manufacturing (Listl et al., 2020). Based on the frequency of updates to the data, the reference architecture differentiates between a dynamic and a static knowledge subgraph. Models related to the product, the manufacturing process, and to experimentation are static in the sense that they do not need to be continuously updated at runtime. These data are embodied in the static subgraph. The dynamic subgraph on the other hand covers operational data accruing in the manufacturing process itself (Ringsquandl et al., 2018, 2017), augmented by added facts from relational machine learning to optimize the interplay between human and AI. This interplay can, e.g., be optimized by proactively giving the operator recommendations for the next manufacturing step or suggesting parameter changes for the machinery to respond to observed product quality deviations, detected by the AI systems.

4.5. Graph-based ML Engine

This component provides relational machine learning capabilities that focus on the application of machine learning methods onto relational or graph-structured data such as knowledge graphs. An essential prerequisite for this is the calculation of graph embeddings

, i.e., the transfer of a graph representation into a vector space. to orchestrate complex machine learning tasks (e.g., via frameworks such as

Kubeflow, TensorFlow Extended, or MLFlow) it provides a framework abstraction layer to make these frameworks accessible through a generic software interface.

4.6. Teaming Engine

The orchestration engine controls the reliable execution of the teaming process, which is defined in the teaming model. This model structures the concrete sequence of activities in the interaction between the AI system and the manufacturing operator. Also, it gives a framework to define which policies are relevant in a specific time period of the manufacturing process and to formalize the ground rules for team interaction, as described by the 4S Interdependence Framework of Johnson and Vera (Johnson and Vera, 2019). For the orchestration of the teaming process, two types of events are processed by the teaming engine: Raw events are extracted from the production line system through IIoT sensors, whereby teaming events are exchanged by the different components after processing. As a result, only teaming events contain contextual data. The orchestration engine can only work with teaming events, since the contextual data must also be evaluated to decide on the next manufacturing activity. The event translation merges several atomic events so that they can be used for decision-making by the orchestration engine.

4.7. Operation Support

The current situation on the shopfloor, i.e., the position of parts and movements of the manufacturing operator, machine settings, and their observed effects on production quality must be taken into account when giving recommendations to the manufacturing operator. Thereby, tracking and scene analysis (TSA) is provided by the situation awareness component, based on media data such as video, imagery or audio recordings. The system’s self diagnosis evaluates the reliability of its recommendations and prematurely alerts upon violations of policies. The interplay of these components, the knowledge graph, and machine learning components for the calculation and selection of recommended actions to the manufacturing operator is controlled by the decision support component.

For the storage of near-realtime data acquired from the production line systems, a time series database is provided. Media data from the TSA system are persisted in the context media database.

4.8. Production Line Systems

Operational data from production and contextual systems, e.g., auxillary systems or manufacturing process control, are acquired from IIoT sensors or from machine-specific implementations, e.g., based on the OPC Unified Architecture (Foundation, 2008). An HMI (Human Machine Interface) is provided to interact with the manufacturing operator, to gather context-dependent feedback about the relevance of given recommendations by the AI. Also, it allows to acquire information about unexpectedly taken actions by the manufacturing operator.

5. Future Work and Conclusion

The presented reference software architecture is framework- and technology agnostic. It shall serve as a blueprint for deriving the software architecture for a particular manufacturing context in a producing company. We use this reference architecture to derive the concrete architecture for the AI-enabled smart manufacturing systems of our three industrial partners in the automotive, energy systems, and precision machining domain. Thus, all three companies have different manufacturing processes in place that result in diverging scalability and interoperability requirements of the software architecture. The validation of the reference architecture will investigate its applicability to different manufacturing contexts, its scalability, and its suitability for validating the compliance of ethical standards during operation. To examine its applicability we will conduct expert interviews with software architects of our industrial partners after having applied the reference software architecture. Likewise, we will conduct interviews with manufacturing operators to assess the suitability of the overall approach and how well it supports the teaming between human and AI in smart manufacturing. Complementary, the quantitative validation will use data acquired from runtime probes to examine the scalability and the consistency among the data shards under heavy load.

6. Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957402.

References

  • S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann (2019) Software Engineering for Machine Learning: A Case Study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300. External Links: Document Cited by: §4.2.
  • A. Arpteg, B. Brinne, L. Crnkovic-Friis, and J. Bosch (2018)

    Software Engineering Challenges of Deep Learning

    .
    In 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 50–59. External Links: Document Cited by: §4.2.
  • J. Bosch, I. Crnkovic, and H. H. Olsson (2020) Engineering AI Systems: A Research Agenda. arXiv:2001.07522 [cs]. Note: arXiv: 2001.07522 External Links: Link Cited by: §3.
  • O. Foundation (2008) What is OPC?. (en-US). Note: Accessed 2021-10-14 External Links: Link Cited by: §4.8.
  • F. Gervits, T. W. Fong, and M. Scheutz (2018) Shared Mental Models to Support Distributed Human-Robot Teaming in Space. In 2018 AIAA SPACE and Astronautics Forum and Exposition, AIAA SPACE Forum. External Links: Link, Document Cited by: §1.
  • T. Hagendorff (2020) The Ethics of AI Ethics: An Evaluation of Guidelines. Minds and Machines 30 (1), pp. 99–120 (en). External Links: ISSN 1572-8641, Link, Document Cited by: §3.
  • HLEG (2019) Ethics guidelines for trustworthy AI | Shaping Europe’s digital future. (en). Note: Accessed 2021-10-10 External Links: Link Cited by: §4.3.
  • IEEE (2019) IEEE Global A/IS Ethics Initiative. (en). Note: Accessed 2021-10-10 External Links: Link Cited by: §4.3.
  • B. Johnson and J. Smith (2021) Towards Ethical Data-Driven Software: Filling the Gaps in Ethics Research amp; Practice. In 2021 IEEE/ACM 2nd International Workshop on Ethics in Software Engineering Research and Practice (SEthics), pp. 18–25. External Links: Document Cited by: §3.
  • M. Johnson and A. Vera (2019) No AI Is an Island: The Case for Teaming Intelligence. AI Magazine 40 (1), pp. 16–28 (en). Note: Number: 1 External Links: ISSN 2371-9621, Link, Document Cited by: §4.6.
  • C. M. Jonker, M. B. van Riemsdijk, and B. Vermeulen (2011) Shared Mental Models. In Coordination, Organizations, Institutions, and Norms in Agent Systems VI, M. De Vos, N. Fornara, J. V. Pitt, and G. Vouros (Eds.), Lecture Notes in Computer Science, Berlin, Heidelberg, pp. 132–151 (en). External Links: ISBN 978-3-642-21268-0, Document Cited by: §1, §3.
  • R. A. Khalil, N. Saeed, M. Masood, Y. M. Fard, M. Alouini, and T. Y. Al-Naffouri (2021) Deep Learning in the Industrial Internet of Things: Potentials, Challenges, and Emerging Applications. IEEE Internet of Things Journal 8 (14), pp. 11016–11040. Note: Conference Name: IEEE Internet of Things Journal External Links: ISSN 2327-4662, Document Cited by: §3.
  • F. G. Listl, J. Fischer, D. Beyer, and M. Weyrich (2020) Knowledge Representation in Modeling and Simulation: A survey for the production and logistic domain. In 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Vol. 1, pp. 1051–1056. Note: ISSN: 1946-0759 External Links: Document Cited by: §4.4.
  • J. Mökander, J. Morley, M. Taddeo, and L. Floridi (2021) Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations. Science and Engineering Ethics 27 (4), pp. 44 (en). External Links: ISSN 1471-5546, Link, Document Cited by: §4.3.
  • A. Nguyen-Duc and P. Abrahamsson (2020) Continuous experimentation on artificial intelligence software: a research agenda. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, New York, NY, USA, pp. 1513–1516. External Links: ISBN 978-1-4503-7043-1, Link, Document Cited by: §3.
  • M. Nickel, K. Murphy, V. Tresp, and E. Gabrilovich (2016) A Review of Relational Machine Learning for Knowledge Graphs. Proceedings of the IEEE 104 (1), pp. 11–33. Note: Conference Name: Proceedings of the IEEE External Links: ISSN 1558-2256, Document Cited by: §1, §3.
  • S. Phuyal, D. Bista, and R. Bista (2020) Challenges, Opportunities and Future Directions of Smart Manufacturing: A State of Art Review. Sustainable Futures 2, pp. 100023 (en). External Links: ISSN 26661888, Link, Document Cited by: §1.
  • S. Pichai (2018) AI at Google: our principles. (en-us). Note: Accessed 2021-10-10 External Links: Link Cited by: §4.3.
  • D. Preuveneers, Y. Berbers, and W. Joosen (2016) SAMURAI: A batch and streaming context architecture for large-scale intelligent applications and environments. Journal of Ambient Intelligence and Smart Environments 8 (1), pp. 63–78 (en). Note: Publisher: IOS Press External Links: ISSN 1876-1364, Link, Document Cited by: §4.
  • Y. J. Qu, X. G. Ming, Z. W. Liu, X. Y. Zhang, and Z. T. Hou (2019) Smart manufacturing systems: state of the art and future trends. The International Journal of Advanced Manufacturing Technology 103 (9), pp. 3751–3768 (en). External Links: ISSN 1433-3015, Link, Document Cited by: §1.
  • M. Ringsquandl, E. Kharlamov, D. Stepanova, M. Hildebrandt, S. Lamparter, R. Lepratti, I. Horrocks, and P. Kröger (2018) Filling Gaps in Industrial Knowledge Graphs via Event-Enhanced Embedding. In Proceedings of ISWC 2018 Posters & Demonstrations, Industry and Blue Sky Ideas Tracks co-located with 17th International Semantic Web Conference (ISWC 2018), External Links: Link Cited by: §4.4.
  • M. Ringsquandl, E. Kharlamov, D. Stepanova, S. Lamparter, R. Lepratti, I. Horrocks, and P. Kröger (2017) On event-driven knowledge graph completion in digital factories. In 2017 IEEE International Conference on Big Data (Big Data), pp. 1676–1681. External Links: Document Cited by: §4.4.
  • C. Rudin (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1 (5), pp. 206–215 (en). Note: Bandiera_abtest: a Cg_type: Nature Research Journals Number: 5 Primary_atype: Reviews Publisher: Nature Publishing Group Subject_term: Computer science;Criminology;Science, technology and society;Statistics Subject_term_id: computer-science;criminology;science-technology-and-society;statistics External Links: ISSN 2522-5839, Link, Document Cited by: §4.3.
  • E. Salas, D. E. Sims, and C. S. Burke (2005) Is there a “Big Five” in Teamwork?. Small Group Research 36 (5), pp. 555–599 (en). Note: Publisher: SAGE Publications Inc External Links: ISSN 1046-4964, Link, Document Cited by: §3.
  • N. Santos, F. Morais, H. Rodrigues, and R. J. Machado (2019) Systems Development for the Industrial IoT: Challenges from Industry R&D Projects. In The Internet of Things in the Industrial Sector: Security and Device Connectivity, Smart Environments, and Industry 4.0, Z. Mahmood (Ed.), Computer Communications and Networks, pp. 55–78 (en). External Links: ISBN 978-3-030-24892-5, Link, Document Cited by: §3.
  • M. Scheutz, S. A. DeLoach, and J. A. Adams (2017) A Framework for Developing and Using Shared Mental Models in Human-Agent Teams. Journal of Cognitive Engineering and Decision Making 11 (3), pp. 203–224 (en). Note: Publisher: SAGE Publications External Links: ISSN 1555-3434, Link, Document Cited by: §1.
  • J. Siebert, L. Jöckel, J. Heidrich, K. Nakamichi, K. Ohashi, I. Namba, R. Yamamoto, and M. Aoyama (2020) Towards Guidelines for Assessing Qualities of Machine Learning Systems. In Quality of Information and Communications Technology - 13th International Conference, QUATIC 2020, Faro, Portugal, September 9-11, 2020, Proceedings, M. J. Shepperd, F. B. e. Abreu, A. R. d. Silva, and R. Pérez-Castillo (Eds.), Communications in Computer and Information Science, Vol. 1266, pp. 17–31. External Links: Link, Document Cited by: §3.
  • T. Trouillon, C. R. Dance, É. Gaussier, J. Welbl, S. Riedel, and G. Bouchard (2017)

    Knowledge graph completion via complex tensor factorization

    .
    The Journal of Machine Learning Research 18 (1), pp. 4735–4772. External Links: ISSN 1532-4435 Cited by: §3.
  • V. Vakkuri, K. Kemell, and P. Abrahamsson (2019) Ethically Aligned Design: An Empirical Evaluation of the RESOLVEDD-Strategy in Software and Systems Development Context. In 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 46–50. External Links: Document Cited by: §3.
  • V. Vakkuri, K. Kemell, M. Jantunen, E. Halme, and P. Abrahamsson (2021) ECCOLA — A method for implementing ethically aligned AI systems. Journal of Systems and Software 182, pp. 111067 (en). External Links: ISSN 0164-1212, Link, Document Cited by: §3.
  • J. Warren and N. Marz (2015) Big Data: Principles and best practices of scalable realtime data systems. 1st Edition edition, Manning (Englisch). Cited by: §4.