What Stands-in for a Missing Tool? A Prototypical Grounded Knowledge-based Approach to Tool Substitution

08/20/2018 ∙ by Madhura Thosar, et al. ∙ Jacobs University Bremen Otto-von-Guericke-University Magdeburg Otto-von-Guericke-Universität Magdeburg 0

When a robot is operating in a dynamic environment, it cannot be assumed that a tool required to solve a given task will always be available. In case of a missing tool, an ideal response would be to find a substitute to complete the task. In this paper, we present a proof of concept of a grounded knowledge-based approach to tool substitution. In order to validate the suitability of a substitute, we conducted experiments involving 22 substitution scenarios. The substitutes computed by the proposed approach were validated on the basis of the experts' choices for each scenario. Our evaluation showed, in 63



There are no comments yet.


page 12

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The sophistication pertaining to tool-use in humans involves not just the dexterity in manipulating a tool, but also the diversity in tool exploitation. The ability to exploit the tools has enabled humans to adapt and thus exert control over an uncertain environment, especially when they are faced with unfavorable situations. For example, if we don’t find a hammer to hammer a nail into a wall, we will use a heel of a shoe or a rock or if a tray is unavailable for serving the drinks, we will use a plate for serving. In situations like these, humans seem to know - either from the past experience or from observations or from the “necessity is the mother of improvisation (invention)” type approach - what kind of object is needed as a substitute.

On the contrary, consider a robot performing a task that involves tool use. When a robot is operating in a dynamic environment, it can not be assumed that a tool required in the task will always be available. In situations like these, an effective way for a robot would be to find an alternative as humans do, for example, use an eating plate for serving, rather than wait until a tray becomes available. This skill is significant when operating in a dynamic, uncertain environment because it allows a robot to adapt to unforeseen situations to a degree. The question is how can a robot determine which object in the environment is a viable candidate for a substitute? A possible approach would be by interacting with an object in a manner missing tool is maneuvered. However, it would be time consuming if a robot interacts with every single object in the environment to determine a viability which makes this approach less practical.

In this prototypical work, we propose a non-invasive approach that identifies viable candidate/s from the existing objects in the environment. This paper makes the following contributions: 1) An approach to create grounded knowledge about objects expressed in terms of their properties (Sec. 5.2), 2) an approach to identify relevant properties of a missing tool and determine a substitute on the basis of them (Sec. 5.3).

2 Related Work

Typically, a substitute for a missing tool is determined by means of knowledge base that provides knowledge about objects and similarity measures to determine the similarity between a missing tool and a potential substitute. In the following, in addition to the approaches to determine a substitute, we also report the literature related to existing knowledge bases developed for robotic applications.

Knowledge Base

We reviewed in Thosar et al. (2018) nine existing knowledge bases namely: KNOWROB Tenorth and Beetz (2009), MLN-KB Zhu, Fathi, and Fei-Fei (2014), NMKB Pineda et al. (2017), OMICS Gupta and Kochenderfer (2004), OMRKF Suh et al. (2007), ORO Lemaignan et al. (2010), OUR-K Lim, Suh, and Suh (2011), PEIS-KB Daoutis, Coradeshi, and Loutfi (2009), and RoboBrain Saxena et al. (2014). The objective was to determine whether these existing knowledge bases contain 1) ontological knowledge about the properties of objects, 2) such knowledge is grounded into robot’s perception, and 3) intra-class variability in a property is modeled instead of expressing the property in a binary form. We gained primarily the following insights which form the basis for our work.

We noted that the majority of the knowledge bases relied on the external human-centric commonsense knowledge bases such as WordNet, Cyc, OpenCyc, and some either relied on the hand-coded knowledge or on the knowledge acquired by human-robot interaction. The main issue, we believe is that, the depth and breadth of the human-centric knowledge base is not observable by a robot in its entirety due to its limited sensing capabilities. This causes a disconnect between human-centric knowledge and robot-centric perception. To deflect this issue, we aim to acquire the robot-centric perceptual data for different properties of objects. Such property data can then be used to generate grounded knowledge about objects (see Sec. 5.1).

Substitution Computation

One of the closest areas that study the usability of an object is affordancs of tools where the primary focus is to examine various functional abilities of an object by exploring what actions can be performed on the object and observing its responses. As such, using a substitute in place of a missing tool can also be seen as transferring of an affordance of the missing tool to the substitute after determining similarity between them.

In Awaad, Kraetzschmar, and Hertzberg (2014), a substitute for a missing tool is inferred on the basis of inheritance and equivalence relations. The work discussed in Agostini et al. (2015) retrieves the knowledge about objects from the ROAR Szedmak, Ugur, and Piater (2014)

relational database and determines a substitute that shares similar affordances. However, in ROAR, the knowledge is acquired either using machine learning techniques requiring training examples or inferred or hand-coded. The work in

Boteanu et al. (2016) uses the ConceptNet where potential candidates are extracted from the knowledge base if they share the same parent with a missing tool for the predetermined relations: has-property, capable-of and used-for. After eliminating irrelevant candidates, a substitute is determined on the basis of the similarity metrics. The approach proposed in Abelha, Guerin, and Schoeler (2016) uses a part-based 3D model and weight of an object to determine the orientation and manipulation of a substitute to be used as a missing tool. In the cases where supervised machine learning technique is used, providing bulk of labeled examples beforehand would not be realistic for a substitution problem scenario. On the other hand, the approaches which rely on existing external knowledge bases are built around the available knowledge in the knowledge bases which does impose some constraints. We circumvents this issue by first identifying what knowledge is generally required to determine a substitute and then build an approach to acquire the required knowledge and compute a substitute on the basis of it.

3 Challenges

How to characterize similarity between a missing tool and a potential substitute: A candidate for a substitute is expected to be similar to a missing tool to some degree to ensure a substitutability. The notion of similarity can be understood in various forms, for instance, a distance between two objects denoted by two points in a multi-dimensional space or two objects belonging to the same cluster or aspects of the objects that are identified as shared. In this work, the question will be addressed in a broader sense: it is not merely about identifying a similar object by deploying some similarity measure, instead, it is about gaining an access to what aspects of the objects were found to be shared between the similar objects.

What kind of knowledge is required to determine the similarity: It has been demonstrated in the literature on tool use in humans and animals alike that in order to use an object in tasks one needs to have knowledge about objects Baber (2003b). Baber in Baber (2003a) also noted that conceptual knowledge about objects is especially desired in tool use where a systematic deliberation is called for. For a robot, the story won’t be much different if it is expected to perform in the real world along side humans. As a consequence, a robot needs conceptual knowledge about an object where the object will not be only a physical entity that is merely to be perceived, but also a concept which consists of distinct characteristics and relations which set each object apart from each other and also similar to each other.

How to acquire the necessary knowledge: The acquisition of such conceptual knowledge is not without challenge. From a robot stand-point, it is a trade-off between what needs to be known and what can be known. The trade-off is a direct consequence of the limited perception capabilities of a robot which often leads to partial understanding of the environment. While deploying a multi-modal perception to extract the required knowledge about objects would be an ideal solution, however, it carries its own set of complexities such as noisy sensors, dynamicity of the environment, complexities of the composition of an object. For this prototypical work, the necessary knowledge is acquired using human-centric as well as machine-centric methods.

How to maneuver a substitute as a missing tool:

Once the substitute has been identified, a robot is expected to use it in place of a missing tool and achieve the same result as the missing tool in the task. The challenge to estimate the maneuver as well as grasping of a substitute is two fold: to determine whether the maneuver and grasping knowledge of a missing tool can be transferred and utilized on a substitute, else, estimate the maneuver and grasping for a substitute such that it can be used as a missing tool in the task.

For this work, we have focused on the first three challenges and have developed a prototypical system called ERSATZ (German word for a substitute or alternative) where the focus is to identify the required knowledge to determine a substitute and develop a system that computes a substitute for a missing tool.

4 Approach

The proposed approach distinguishes a tool from a substitute where a tool is defined as an artifact that is designed, manufactured and maneuvered in accordance with its designated purpose in the tasks such as hammer for hammering, tray for serving etc., while a substitute is seen as an extension of a missing tool. Within the context of a designated purpose, the relationship between a tool and a substitute is symmetric, for instance, for hammering, a hammer can be replaced by a heeled shoe and vice versa. However, it may not always be the case once you step outside the context, for instance, a hammer can not replace a heeled shoe. Our research work, therefore, focuses on searching for a substitute for a conventional tool required in the ongoing task as opposed to determining a substitute for itself.

Consider a scenario in which a robot has to choose between a plate and a mouse pad as an alternative for a tray. A tray can be defined as a rigid, rectangular, flat, wooden, brown colored object while a plate can be defined as a rigid, circular, semi-flat, white colored object and a mouse pad as soft, rectangular, flat, leather-based object. Bear in mind, however, that some properties are more relevant than others with respect to the designated purpose of the tool. For a tray whose designated purpose is

to carry, rigid and flat are more relevant to carry than a material or a color of a tray. Consequently, to find the most appropriate substitute, the relevant properties of the unavailable tool need to correspond to as large a degree as possible to the properties of the possible choices for a substitute.

The proposed approach performs conceptual knowledge-driven computation to identify the relevant properties of the missing tool and determines the most similar substitute on the basis of those properties. Besides identifying the most similar object as a substitute for a missing tool, the proposed approach grants an explicit access to the relevant properties of the missing tool which carries twofold advantages: firstly, knowing which properties are primarily required in the potential substitute narrows down the search space and secondly, in case of an unknown object instance, only the relevant properties will have to be learned to determine a substitute.

The conceptual knowledge considered in this work primarily involves properties of the objects. The properties considered are divided into physical and functional properties where physical properties describe the physicality of the objects such as rigidity, weight, hollowness while the functional properties ascribe the (functional) abilities or affordances to the objects such as containment, blockage, support. The functional properties in the proposed approach play a primary role in identifying the relevant properties of a missing tool (see Sec. 5.3).

The functional properties considered in this work are derived from the theory of image schemas Gärdenfors (1987) which has its roots in cognitive linguistics. According to Kuhn (2007) image schemas are patterns abstracted from spatio-temporal experiences. Essentially image schemas capture recurrent patterns that emerge from our perceptual and bodily interactions with the environment. Since some of these patterns are posited on the operational abilities of objects Kuhn postulated in Kuhn (2007) that affordances a.k.a. functional properties Gärdenfors (1987) for the spatio-temporal processes can be derived from image schemas. For example, the containment schema suggest an object’s ability to contain something or the support schema indicates an object’s ability to hold up something or the blockage refers to the ability of an object to block or obstruct the movement of an other object. Currently, the proposed system is restricted to three functional properties based on image schemas: containment, support and blockage. While the functional properties as well as the designated purpose can both be identified as affordances, the proposed approach is built by hypothesizing that the functional properties are building blocks upon which designated purposes of tools rest.

5 Methodology

5.1 Knowledge Acquisition

Our ultimate objective is to acquire machine centric data from which property specific data can be extracted. Such property data will then be used to generate grounded knowledge about objects. As a first step, our initial property acquisition focuses on the composite of a machine-centric and a human-centric method. In the machine-centric approach, geometrical properties are acquired using a non-invasive vision-based technique while non-geometric properties are acquired by sampling from the data from the expert generated intuitive model for the properties.

Machine Generated Properties: In this paper, we introduce a state-of-art data-driven approach that unsupervisedly conceptualizes shape according to commonalities within object point clouds which is discussed in detail in our work Mueller and Birk (2018). As a result of the process, a set of shape concepts is generated which concept responses for an unknown object are used in the knowledge base as machine-generated geometric object properties.

In our previous work on shape concept learning Mueller and Birk (2018), raw sensor information in form of point clouds is abstracted to a symbolic level in which point cloud segments Mueller and Birk (2016) may represent meaningful shape components in a symbolic space Mueller, Pathak, and Birk (2014). Therein we introduce a hierarchical learning procedure that leads to symbols which are gradually organized to reflect generic-to-specific facets of shape components and can be subsequently used as building blocks that constitute objects (see A in Fig. 1).

Figure 1: Illustration of the object shape conceptualization approach Mueller and Birk (2018). Concepts are randomly colored.

An object shape representation is introduced that gradually encodes observed objects symbol compositions (see B in Fig. 1): from local components to component groups that may represent object parts or objects as a whole. The proposed shape representation incorporates aspects of exemplar, respectively, prototype theory since we believe that the richness of a prototype provides an unaltered perspective on the characteristics of object instances. Based on the proposed symbolic shape representation we analyze topology and structure within the encoded symbol compositions in order to discover persistent patterns that may represent shape concepts.

We introduce an iterative filtering process Mueller and Birk (2018) to associated instances to groups which may represent shape concepts (see C in Fig. 1). Given the set of learned concepts, for an unknown object, concept responses are retrieved (see D in Fig. 1) and exploited as machine-generated geometric object property values in our tool-substitution scenario. Note that in our tool-substitution scenario, concepts are learned from unlabeled object instances of the Object Discovery Dataset(ODD) Mueller and Birk (2016)

; the ODD provides a variety of objects from

teddy bears over flash lights to shoes which facilitates an expressive concept generation.

Human Generated Properties The geometric properties alone offer a very limited scope of the physicality as well as the functionality of an object. Therefore, to compensate the gap, we also considered non-geometrical properties such as weight, rigid, hollowness as physical and support, blockage, containment as functional. Note that, in general, these properties are challenging and cumbersome to extract solely from non-invasive visuoperceptual approaches. Consequently, extracting such properties via multi-modal or manipulation capabilities is needed, but this is beyond the scope of this paper. In the generation process, a set of labeled prototype objects selected from the Washington dataset (see Table 1) were taken into account. The distribution of each property for particular object labels (cf. Table 1) was approximated by an expert to resemble the scope for the variations in the values of the property in general. Consequently, given an object and its label, a sample value was drawn from the a-priori generated property distribution.

5.2 Knowledge about Objects

Knowledge about objects is spread across three levels: the first level consists of the data about the machine-generated as well as human-generated properties, the second level consists of qualitative knowledge about individual object instances, while the third level consists of the aggregated qualitative fuzzy knowledge about respective classes of object instances. The fuzzy formalism is used to model the intra-class variations in the objects. In the following, we discuss the formal description of the methodology deployed to create grounded knowledge about objects.

Consider O as a given set of object class labels where (by abuse of notation) each object class is identified with its label. Let each object class be a given set of its instances. Let be a union of all object classes such that . Let P and F be the given sets of physical properties’ labels and a set of functional properties’ labels respectively. By abuse of notation, each physical and functional property is identified with its label. For each physical property as well as for a functional property , sensory data is acquired from each object instance . Let and represent sets of number of extracted sensory values from number of object instances for a physical property and a functional property respectively .

Sub-categorization - From Continuous to Discrete

The sub-categorization process is performed to form (more intuitive) qualitative measures to represent the degree with which a property is reflected by an object instance. It is the first step in creating symbolic knowledge about object classes where the symbols representing the qualitative measures of a physical or a functional property reflected in an object instance are generated unsupervisedly by a clustering mechanism. A qualitative measure of a physical property is referred to as a physical quality and that of a functional property as a functional quality.

In this process, and representing measurements of a physical property and a functional property respectively extracted from number of object instances is categorized into a given number of discrete clusters using a clustering algorithm. Let and be partitions of the sets and after performing clustering on them. Let and be the sets of labels, expressing physical qualities and functional qualities, generated for a physical property and a functional property respectively. Given the label for a property, the quality labels are generated by combining a property label and a cluster label (created by the clustering algorithm). For instance, the quality labels for a property are represented as . At the end of the sub-categorization process, the clusters are mapped to the generated symbolic labels for qualitative measures.

Note that the number of clusters essentially describes the granularity with which each property can qualitatively be represented. The higher number of clusters suggest that an object is described in a finer detail which may obstruct the selection of a substitute since it may not be possible to find a substitute which is similar to a missing tool down to the finer details. For example, in , size is a physical property and small, medium, big, bigger are its physical qualities. The semantic terms given above are meant for the readers to understand the qualitative measures of the properties.

Attribution - Object Instance Knowledge

The attribution process generates knowledge about each object instance by aggregating all the physical and functional qualities assigned to the object instance by the sub-categorization step. In other terms, the knowledge about an instance consists of the physical as well as functional qualities reflected in the instance. Let and be the families of sets containing the physical quality labels and the functional quality labels for each physical property and functional property respectively. Thus, each object instance is represented as a set of all the physical as well as functional qualities attributed to it which are expressed by a symbol as: For example, knowledge about the instance of a plate class can be given as, , , where medium is a physical quality of size property, harder is a physical quality of rigidity property and can_support is a functional quality of support property.

Conceptualization - Knowledge about Objects

The conceptualization process aggregates the knowledge about all the instances of an object class. The aggregated knowledge is regarded as conceptual knowledge about an object class.

Let be a knowledge base about object classes where each object class . Given the knowledge about all the instances of an object class , in the conceptualization process, the knowledge about the object class is expressed as a set of tuples consisting of a physical or a functional quality and its proportion (membership) value in the object class. A tuple is expressed as where and a proportion value is calculated using the following membership function: . The proportion value allows to model the intra-class variations in the objects.

For example, knowledge about object class table can be expressed as: {plate, harder, 0.6, plate, light_weight, 0.75, plate, less_hollow, 0.67, plate, hollow, 0.33, plate, more_support, 0.71}, where the numbers indicate that, for instance, physical quality harder was observed in 60% instances of object class plate. At the end of the conceptualization process, conceptual knowledge about an object class is created which is represented in a symbolic fuzzy form and grounded into the human-generated or machine-generated data about the properties of objects. The knowledge about objects is then used to determine a substitute from the existing objects in the environment.

Conceptualization - Knowledge about Functional Properties

In addition to conceptual knowledge about objects, Conceptualization process also creates knowledge about functional quality, termed as a function model, by associating the occurrence of physical qualities in an object instance with the occurrence of a functional quality in the instance and aggregating the result of such concurrent occurrences. The role of a functional model is discussed later in the section 5.3. Given the knowledge about the object instances, a function model of a functional quality is expressed as a set of tuples containing a functional quality , a physical quality and a proportion value . A tuple is represented as where and a proportion value is computed as, For example, a function model for a functional quality more_support is given as, { more_support, harder,0.8, more_support, softer, 0.2 where the number indicates that, for instance, functional quality and a physical quality co-occurred in the knowledge about the object instances % of the time.

5.3 Reasoner

Figure 2: A typical process flow to determine a substitute for a missing tool from the available objects.

Fig. 2 illustrates a process flow consisting of the primary operations involved in determining a substitute. The flow offers an approximated aerial view for the prototypical model of ERSATZ. When ERSATZ is queried to find a substitute for a missing tool x from the set of available objects Y the system checks if the substitution model for x exists in the knowledge base. If the substitution model does not exist, then the reasoner computes the relevant functional and physical properties of the queried tool.

Representative Models

A representative physical model and a representative functional model of an object consists of the physical or functional qualities, respectively, that are regarded as representative qualities of the object class, while the qualities which do not fall under representative qualities are regarded as exceptional or uncommon qualities.

Let be an object class of a missing tool and let is a representative model threshold which qualifies a physical or a functional quality as stereotypical or representative to the object class . is called as a representative physical model of an object class such that and is called as a representative functional model of an object such that . Similarly, let be a function model of functional quality , then is called as a representative physical model of a functional quality such that

Relevant Qualities

Due to the abstract nature of an image schema and by extension a corresponding functional property, it can subsume various purposes of objects, for example, a functional property support which can subsume the purposes place on, sit on and serve on of the a table, a chair and a tray respectively. It is suggested in  Baber (2003c) that a certain assemblage of physical properties are essential prerequisites to enable a functional property. Thus, it can be assumed that by knowing the relevance of one functional property can help identify the relevant physical properties of different objects which are used for different purposes.

The relevance of a representative functional quality is decided by examining whether the physical characterization of the function model of the representative functional quality of a tool are in a close proximity to the physical characterization of a representative physical model of the tool. The close proximity between a functional quality and the object class of the tool is determined using Jaccard Index. Jaccard Index determines a similarity and dissimilarity between the two sets A and B where the similarity is calculated by dividing the magnitude of the intersection of A and B by the magnitude of the union of A and B.

Let and be the representative physical models of an object class of the missing tool and of a function model of a representative functional quality of the object class respectively. Let be a Minimum Similarity Tolerance threshold for similarity. Then, Jaccard Index of and is computed as: . A representative functional quality of an object class is regarded as relevant if . Let be a set of all relevant functional qualities of an object class . Let be a representative physical model of a function model of a relevant functional quality . Let be a representative physical model of . Then, the relevant physical qualities of an object class , expressed by a set .

Reasoning about a Substitute

Let be an object class of a missing tool and let be an object class of a possible candidate for a substitute. Let be a set of relevant physical qualities of and let be a representative physical model of . Let be a Minimum Similarity Tolerance threshold for similarity. The substitutability of a candidate is determined by measuring the similarity between and using Jaccard’s Index. is termed as a substitute, expressed as , if , else it is regarded as not a substitute and expressed as . Given the set of relevant physical qualities , the set of relevant functional qualities and a positive substitute , and a negative substitute , a substitution model of is expressed as a tuple: . The knowledge about object is then extended in to accommodate its substitution model.

6 Experimental Evaluation

The objective of the experimental evaluation of ERSATZ is to validate the suitability of the substitutes computed by ERSATZ by comparing the results with that of human experts. For the experimental evaluation, we used the images from the Washington Dataset Lai et al. (2011) to generate human-based and machine-based properties. Around object categories were selected and for each category, we selected random images from all the given instances of the category leading up to total of images. Table 1 illustrates the number of images selected from each category. For the experiment, we generated queries based on object categories. Each query consisted of a missing tool and randomly selected objects from which a substitute was to be selected. We gave queries, to human experts and asked them to select a substitute in each query. The distribution of the human selections for each scenario is illustrated in Fig. 3(a)

. Similarly, the queries were run on ERSATZ with the following (heuristically determined) optimal values of the target parameters: i) Number of machine-generated properties is set to

(Sec. 5.1), ii) Number of clusters to (Sec. 5.2), iii) Representative threshold (Sec. 5.3) and Minimum Similarity Tolerance (Sec. 5.3) to .







cereal box

coffee mug


food bag

food box

food can

food cup

food jar

hand towel







soda can


water bottle

























per Inst.























# 35 30 30 32 30 32 30 32 36 28 30 30 30 30 30 30 30 35 30 30 36 36
Table 1: Number of scans (#) per category () of the Washington RGBD dataset Lai et al. (2011).
(a) Human expert selection distributions
(b) ERSATZ selections with similarity to the missing tool
Figure 3: Substitution results w.r.t. human expert selection distribution and ERSATZ similarity responses. Note that, gray cells correspond to object categories which are not available in the respective query, cells marked with represents substitutes selected by experts and ERSATZ.

The results of both experiments were plotted as a heat map where the y-axis shows missing tools and x-axis shows the available objects illustrated in Fig. 3. The grayed cells mean the corresponding object categories were not available in the respective query. The cells that are marked with represents substitutes selected by experts and ERSATZ. Out of scenarios, ERSATZ and the experts identified the same substitutes in scenarios (%).

7 Future Work

The paper presents a prototypical system to determine a substitute for a missing tool using the grounded knowledge about objects. The approach has drawn inspiration from symbol grounding, the theory of affordances and the theory of image schemas to represent the grounded knowledge and to determine a substitute. This is an ongoing research with a focus on the following aspects.

Our immediate goal focuses on the fuzzification of the clustering method and the reasoning method to combat the migration of the data points within clusters. Moreover, we have derived three functional properties, namely, contain, support, block from the image schemas Containment, Support and Blockage respectively. However, further investigation is needed to formalize the identification of additional functional properties to be derived from the existing image schema.

Figure 4: Multi-layered dataset to build a robot-centric grounded knowledge about objects.

For robot-centric property acquisition, we are currently developing a framework that allows a robot to extract properties of individual objects and build a knowledge base in a bottom-up manner such that the knowledge about properties of objects is constructed on the basis of what is sensed (see Fig. 4). We have proposed the preliminary framework in Jäger et al. (2018).


  • Abelha, Guerin, and Schoeler (2016) Abelha, P.; Guerin, F.; and Schoeler, M. 2016. A model-based approach to finding substitute tools in 3D vision data. Proceedings - IEEE International Conference on Robotics and Automation 2016-June.
  • Agostini et al. (2015) Agostini, A.; Aein, M. J.; Szedmak, S.; Aksoy, E. E.; Piater, J.; and Worgotter, F. 2015. Using structural bootstrapping for object substitution in robotic executions of human-like manipulation tasks. IEEE International Conference on Intelligent Robots and Systems 2015-Decem:6479–6486.
  • Awaad, Kraetzschmar, and Hertzberg (2014) Awaad, I.; Kraetzschmar, G. K.; and Hertzberg, J. 2014. Challenges in Finding Ways to Get the Job Done. Planning and Robotics (PlanRob) Workshop at 24th International Conference on Automated Planning and Scheduling.
  • Baber (2003a) Baber, C. 2003a. Introduction. In Cognition and Tool Use. Taylor and Francis. chapter 1, 1–15.
  • Baber (2003b) Baber, C. 2003b. Cognition and Tool Use. Taylor and Francis.
  • Baber (2003c) Baber, C. 2003c. Working With Tools. In Cognition and Tool Use. 51–68.
  • Boteanu et al. (2016) Boteanu, A.; St. Clair, A.; Mohseni-Kabir, A.; Saldanha, C.; and Chernova, S. 2016. Leveraging Large-Scale Semantic Networks for Adaptive Robot Task Learning and Execution. Big Data 4(4):217–235.
  • Daoutis, Coradeshi, and Loutfi (2009) Daoutis, M.; Coradeshi, S.; and Loutfi, A. 2009. Grounding commonsense knowledge in intelligent systems. Journal of Ambient Intelligence and Smart Environments 1(4):311–321.
  • Gärdenfors (1987) Gärdenfors, P. 1987. Cognitive semantics and image schemas with embodied forces. Image (Rochester, N.Y.) 1–16.
  • Gupta and Kochenderfer (2004) Gupta, R., and Kochenderfer, M. J. 2004. Common Sense Data Acquisition for Indoor Mobile Robots. In

    Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on Innovative Applications of Artificial Intelligence

    , 605–610.
  • Jäger et al. (2018) Jäger, G.; Mueller, C. A.; Thosar, M.; Zug, S.; and Birk, A. 2018. Towards Robot-Centric Conceptual Knowledge Acquisition. In Robots that learn and reason in IEEE/RSJ International Conference on Intelligent Robots and Systems.
  • Kuhn (2007) Kuhn, W. 2007. An Image-Schematic Account of Spatial Categories. Spatial Information Theory 152–168.
  • Lai et al. (2011) Lai, K.; Bo, L.; Ren, X.; and Fox, D. 2011. A Large-Scale Hierarchical Multi-View RGB-D Object Dataset. In IEEE International Conference on Robotics and Automation (ICRA), 1817–1824.
  • Lemaignan et al. (2010) Lemaignan, S.; Ros, R.; Mösenlechner, L.; Alami, R.; and Beetz, M. 2010. ORO, a knowledge management platform for cognitive architectures in robotics. IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010 - Conference Proceedings (April):3548–3553.
  • Lim, Suh, and Suh (2011) Lim, G. H.; Suh, I. H.; and Suh, H. 2011. Ontology-based unified robot knowledge for service robots in indoor environments. IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans 41(3):492–509.
  • Mueller and Birk (2016) Mueller, C. A., and Birk, A. 2016. Hierarchical Graph-Based Discovery of Non-Primitive-Shaped Objects in Unstructured Environments. In International Conference on Robotics and Automation.
  • Mueller and Birk (2018) Mueller, C. A., and Birk, A. 2018. Conceptualization of Object Compositions Using Persistent Homology. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
  • Mueller, Pathak, and Birk (2014) Mueller, C.; Pathak, K.; and Birk, A. 2014.

    Object shape categorization in rgbd images using hierarchical graph constellation models based on unsupervisedly learned shape parts described by a set of shape specificity levels.

    In International Conference on Intelligent Robots and Systems.
  • Pineda et al. (2017) Pineda, L. A.; Rodríguez, A.; Fuentes, G.; Rascón, C.; and Meza, I. 2017. A light non-monotonic knowledge-base for service robots. Intelligent Service Robotics 10(3):159–171.
  • Saxena et al. (2014) Saxena, A.; Jain, A.; Sener, O.; Jami, A.; Misra, D. K.; and Koppula, H. S. 2014.

    RoboBrain: Large-Scale Knowledge Engine for Robots.

    arXiv 1 – 11.
  • Suh et al. (2007) Suh, I. H.; Lim, G. H.; Hwang, W.; Suh, H.; Choi, J. H.; and Park, Y. T. 2007. Ontology-based multi-layered robot knowledge framework (OMRKF) for robot intelligence. IEEE International Conference on Intelligent Robots and Systems (October):429–436.
  • Szedmak, Ugur, and Piater (2014) Szedmak, S.; Ugur, E.; and Piater, J. 2014. Knowledge Propagation and Relation Learning for Predicting Action Effects. In IEEE/RSJ International Conference on Intelligent Robots and Systems.
  • Tenorth and Beetz (2009) Tenorth, M., and Beetz, M. 2009. KNOWROB- Knowledge Processing for Autonomous Personal Robots. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 4261–4266.
  • Thosar et al. (2018) Thosar, M.; Zug, S.; Skaria, A. M.; and Jain, A. 2018. A Review of Knowledge Bases for Service Robots in Household Environments. In 6th International Workshop on Artificial Intelligence and Cognition.
  • Zhu, Fathi, and Fei-Fei (2014) Zhu, Y.; Fathi, A.; and Fei-Fei, L. 2014. Reasoning About Object Affordance in a Knowledge Based Representation.

    European Conference on Computer Vision