Governance of Autonomous Agents on the Web: Challenges and Opportunities

The study of autonomous agents has a long tradition in the Multiagent Systems and the Semantic Web communities, with applications ranging from automating business processes to personal assistants. More recently, the Web of Things (WoT), which is an extension of the Internet of Things (IoT) with metadata expressed in Web standards, and its community provide further motivation for pushing the autonomous agents research agenda forward. Although representing and reasoning about norms, policies and preferences is crucial to ensuring that autonomous agents act in a manner that satisfies stakeholder requirements, normative concepts, policies and preferences have yet to be considered as first-class abstractions in Web-based multiagent systems. Towards this end, this paper motivates the need for alignment and joint research across the Multiagent Systems, Semantic Web, and WoT communities, introduces a conceptual framework for governance of autonomous agents on the Web, and identifies several research challenges and opportunities.



page 3


Intelligent Software Web Agents: A Gap Analysis

Semantic web technologies have shown their effectiveness, especially whe...

Challenges and Opportunities in Integrated Space-Terrestrial Internet of Things

Large geographical regions and communities remain uncovered by terrestri...

Specifying Autonomy in the Internet of Things: The Autonomy Model and Notation

Driven by digitization in society and industry, automating behavior in a...

Towards the Internet of Robotic Things: Analysis, Architecture, Components and Challenges

Internet of Things (IoT) and robotics cannot be considered two separate ...

Semantic Web Environments for Multi-Agent Systems: Enabling agents to use Web of Things via semantic web

The Web is ubiquitous, increasingly populated with interconnected data, ...

A Requirements Engineering Technology for the IoT Software Systems

Contemporary software systems (CSS), such as the internet of things (IoT...

Analysis for the Overwhelming Success of the Web Compared to Microcosm and Hyper-G Systems

Microcosm, Hyper-G, and the Web were developed and released after 1989. ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Over the last two decades, the Web has evolved extensively in response to a variety of different requirements. From originally providing a distributed information dissemination architecture, it has encompassed support for publication, discovery, consumption and aggregation of information, knowledge, and services, thereby interconnecting the digital, social and physical worlds. The Web’s ubiquity, as well as the simplicity of its underlying communication protocols has resulted in it becoming the de facto standard for communication between services, and more recently, connected things. With the rise of the Internet of Things (IoT), the combination of the Web of Things (WoT) (as an extension of the IoT with metadata expressed in Web standards), traditional web services, and a knowledge dissemination infrastructure that is both machine navigable and machine understandable, has facilitated a new generation of applications that utilise the Web.

Berners-Lee et al. (2001) outline how autonomous agents could comprehend and exploit this machine-readable knowledge to achieve a variety of tasks. Thus, the notion of autonomy provides a framework whereby individual agents (e.g., those representing or controlling services, things, or applications) may plan, collaborate, and cooperate to achieve complex but disparate goals. Such multiagent systems avoid centralised control, which is the bane of business process management (Singh and Huhns, 2005). By seeking mutually beneficial interactions, agents of heterogeneous construction (potentially originating from different developers) can evolve a mutually supportive economy across the Web, performing a multitude of tasks for Web users. However, to achieve this notion of collaborative agents that use the Web infrastructure, it is crucial to consider a governance perspective, which defines how agents should act in a given situation (also considering the consequences of their potential actions) and defines how frameworks that govern communities of agents should be designed, interoperate, and evolve. This perspective is of particular importance for the Web, where usage may cross social contexts and jurisdictions, and where no centralised control over the different agents is possible. Indeed, the need for intelligent system governance is, at the time of writing, a focus point of legislative and regulatory efforts; e.g., by the European Commission 

(High-Level Expert Group on Artificial Intelligence (AI HLEG),


Therefore, what is needed is a new governance framework supported by a review of the related literature on the use of norms, policies, and preferences for autonomous normative agents, as well as contextualising these with respect to the notion of the Web (of Things). Towards this end, this paper makes three main contributions. Firstly, it motivates the need for norms, policies and preferences for autonomous agents on the Web by means of a simple motivating scenario. Secondly, it proposes a new governance conceptual framework and gives an overview of the state of the art on norms, policies, and preferences for autonomous normative agents (restricted to those efforts that provide the theoretical background to our proposed framework). Finally, it identifies several challenges and opportunities, for the MAS, Semantic Web and WoT communities, underlining the need for better integration and joint research across the different communities. Each challenge is motivated by a concise review of the state of the art, followed by several opportunities for future investigation. For each of the identified challenges, we discuss its maturity in terms of research and technological approaches, ranging from nascent solutions to those that have received community adoption, whereas the opportunities take the form of open research questions that need to be explored.

The remainder of the paper is structured as follows: we start by outlining our motivating use case scenario (Section 2) and presenting the relevant background (Section 3). A conceptual framework is then proposed (Section 4), accompanied by an instantiation based on our use case scenario (Section 5), after which we subsequently identify several challenges and opportunities (Section 6). We then conclude by proposing a research roadmap for the governance of autonomous agents on the web (Section 7).

2. Motivating Scenario

The synergy between autonomous agents that leverage Web technologies, and the resources (i.e., things, services, information) that they can exploit to achieve their goals can be illustrated through a motivating scenario that demonstrates the need for governance through the use of norms, policies, and preferences. Consider a scenario whereby there is a vaccination roll-out (for example, for the COVID-19 pandemic), where patients who request vaccinations may have differing personal circumstances. For example, John, the patient in Figure 1, may ask to be vaccinated early as he is the care giver for a vulnerable member of his family. As the demand for vaccines outstrips supply, policies exist that determine vaccination eligibility. Furthermore, as vaccines are available from different manufacturers (e.g., AstraZeneca and Pfizer-BioNTech) and can be of different types (e.g., mRNA or inactivated vaccines), these vaccination policies may vary depending on the recipient’s personal health record and/or their preferences, as well as vaccine availability.

Patients may be registered to different clinics or health centres that follow local or national policies or guidance on health care. In this case, John is registered at a clinic in his country (labelled State B in Figure 1), but has a preference for vaccination near his current residential address in State A. Each country or state can be seen as having an organisation of different health centres (clinics, hospitals, and vaccination centres), following their own national health policy that prescribe a specific specification/format for patient medical records, which may be held under disparate data models and access policies. Patient medical records are available (subject to appropriate authorisation) via web services using secure protocols across the web infrastructure (Tanwar et al., 2020), and are encoded using established medical ontologies and vocabularies to facilitate record exchange within and across different national health organisations.

Figure 1. Organisations, agents, things, and services in the scenario.

Vaccination centres store batches of vaccines within one or more temperature-controlled vaccine storage systems, where each storage system is responsible for both inventory management and the dispensation of the different COVID-19 vaccine batches from a specialised cold store via a robotic arm. The release and retrieval of vaccine batches is guarded by policies that must be satisfied to ensure appropriate use by authorised personnel (i.e., the vaccine guard in Figure 1). Once a batch of vaccines has been released, the vaccine doses should be used within a given time-frame to avoid spoilage and wastage, as they have a short shelf-life once thawed. Furthermore, a scheduling system determines which patients can be vaccinated in a given time-slot, based on vaccination demand and patient requirement (determined by the current vaccination policy that may change frequently). This scheduling system should ensure that no vaccines are wasted, whilst ensuring that the policies determining which patients can receive which vaccines is adhered to. Thus, the vaccination centre could be considered as an organisation that coordinates and exploits a variety of disparate information technology (IT) systems integrated through a Web infrastructure, including data management, scheduling, patient-facing services, and IoT-based physical assets such as the robot arm and the automated vaccine stores. Typically, however, the task of orchestrating and using these different systems requires costly and time-consuming human intervention. Finally, once a vaccine has been administered, the patient’s medical records should be updated, and the patient should be able to prove their vaccination status if required (e.g., using a vaccine passport (Eisenstadt et al., 2020)). The vaccine records should ideally be resilient to forgery whilst being privacy preserving and easy to administer (Eisenstadt et al., 2020); thus they may utilise a passport mechanism that itself exploits web-based resources such as verifiable credentials111, decentralised data platforms (Verborgh, 2018), blockchains (Tanwar et al., 2020), etc.

This scenario raises challenges due to the decentralised and dynamic characteristics of the involved organisations, policies, services, and stakeholders. Patients can request vaccination based on their interpretation of eligibility, which should then be validated by the vaccination centre. The handling of requests may require the collection of patient data from multiple sources and the mapping to a shared data model. The vaccination eligibility policy can change frequently due to, for example, the emergence of a new variant of concern, that may accelerate the need for vaccinating a specific population cohort or demographic. Changes to vaccination administration guidance may prioritise the use of certain types of vaccine over others for specific sub-groups (e.g., prioritising Pfizer-BioNTech over AstraZeneca, where possible, for certain patients based on medical risk assessments, or prohibiting certain vaccines for users where safety data is not available). Thus, the verification of vaccination eligibility for patients may rely on the aggregation of multiple policies, and on resolving inconsistencies between them. A further challenge involves ensuring that the process for adhering to the national prioritisation criteria is fair and transparent.

Additional legal and ethical challenges arise when considering the complete socio-technical system, including electronic health record access (Kafali et al., 2020) and supply chains (Singh et al., 2009). Finally, vaccination scheduling needs to take into account patient availability (to avoid no-show cases and thus avoid vaccine wastage), as well as stock availability. Scheduling is therefore a collaborative process involving factors such as the vaccination centre capacity, vaccine availability, and patient availability. However, availability data may be distributed across multiple sources and, for privacy reasons, cannot be held centrally.

This scenario underlines the need for systematic and scalable approaches for the governance of the different IT systems and IoT-based physical assets, taking into account the need to operate under different governance institutions, as well as interact across organisational boundaries (e.g., between countries). Such interactions must comply with applicable norms and policies encountered at different stages of the vaccination roll-out. For example, the European Commission recently proposed a Digital Green Certificate, recognised by all EU member states, that facilitates the safe free movement of citizens within the EU during the COVID-19 pandemic.222

Given the intrinsic openness of the Web, coupled with the fact that autonomous agents can act on behalf of both patients and medical practitioners that need access to critical medical applications, the need for regulation, security, and privacy are of utmost importance. Additionally, there is a need to facilitate coordination between stakeholders and ensure that relevant regulatory requirements are adhered to throughout.

3. Background

The vaccine administration scenario detailed in Section 2 relies on the availability of a uniform access layer that sits on top of several different systems (e.g., data management, services, and IoT platforms). The Web provides the necessary infrastructure to integrate and make accessible all of these systems, effectively becoming an application architecture for the proposed MAS (Ciortea et al., 2019), on top of which autonomous agents may interact and cooperate to achieve common goals. In the following subsections, we present the relevant background in multiagent systems, the Semantic Web, and the WoT, followed by a discussion of the related work in norms, policies and preferences, with a focus on the governance of autonomous agents, both within and spanning those communities.

3.1. Multiagent Systems

A multiagent system is composed of a (dynamic) set of agents interacting inside a shared, possibly distributed, environment which itself comprises a dynamic set of artefacts. Agents are goal-oriented autonomous entities, encapsulating a logical thread of control, that pursue their tasks by communicating with other agents and by perceiving and acting upon artefacts within the environment. In essence, a MAS addresses the challenges of how agents may coordinate their efforts and cooperate in light of their autonomy (Weiss, 1999). Artefacts model any kind of (non-autonomous) resource or tool that agents can use and possibly share to achieve their goals. An agent perceives the observable state of an artefact, reacts to events related to state changes, and performs actions that correspond to operations provided by the artefact’s interface. The coordinated and organised activities taking place in the system result from the concurrent and complex tasks handled by groups of agents interacting with each other, or acting within an environment. Such activities may lead to recurrent patterns of cooperation captured by agent organisations. Changes in the state of the environment may also lead agents to react and possibly affect the state of the organisation.

Research into multiagent systems has led to a number of concrete programming models.333Refer to the proceedings of the EMAS or PAAMS series for broad overviews. These models444The models presented here reflect the relevant state of the art with respect to different MAS and are by no means exhaustive.are concerned with agent-oriented programming (Bordini et al., 2007), interaction and protocol languages (Singh, 2011), environment infrastructures (Weyns et al., 2007), and agent organisation model and management systems (Ferber et al., 2004). The results produced so far have clearly demonstrated the importance of these concepts and abstractions for the development of multiagent applications. Additionally, a variety of languages, tools, and platforms for agent-oriented programming (MAOP) have been developed and application success stories exists (e.g., (De la Prieta et al., 2019)). This type of research is often referred to under the umbrella of Engineering Multiagent Systems (EMAS). An overview and a comparative analysis of several prominent MAOPs can be found in (Kravari and Bassiliades, 2015). One of the most prominent underlying architectures used by many agent-oriented programming systems is the Belief-Desire-Intention (BDI) architecture, which models: knowledge (i.e., beliefs) that the agent knows about, either through observation of the environment or interaction with other agents; goals (i.e., desires) that the agent would like to bring about; and goals and plans of action (i.e., intentions) that the agent is currently focused on.

From an agent development environment perspective, the Jade platform (Bellifemine et al., 2007) provides a variety of behaviours (one-shot, cyclic, contract net) and is still available, although the last release dates back to 2017. Although Jade does not directly provide support for BDI-based agents, they can be added through extensions such as Jadex (Braubach et al., 2005). Jack (Busetta et al., 2019)

is an example of a closed source BDI architecture, whereas the practical Agent Programming Language (2APL) is another open source language that retains BDI semantics

(Dastani, 2008). GOAL (Hindriks and Dix, 2014) offers a further BDI architecture which is actively maintained, whereas SPADE555 is a recently introduced Python-based BDI platform. The JaCaMo MAOP framework, based on the JaCaMo conceptual meta-model (Boissier et al., 2020), offers first-class abstractions to program the agents working environment and their organisation, in addition to offering the Jason interpreter for the BDI-based AgentSpeak language (Bordini et al., 2007).

Whilst MAOP is thriving within the academic community, industrial adoption of MAOP technologies is in its infancy, and standardisation efforts such as FIPA (FIPA, 2003) (that superseded KQML) have received little attention in recent years (Mascardi et al., 2019).

3.2. Agents and the Semantic Web

Attempts to tightly integrate autonomous agents and Web technologies date back to the vision of the Semantic Web of the early 2000s. Berners-Lee et al. (2001) originally envisioned “a web of data that can be processed directly and indirectly by machines”, in which intelligent agents act on behalf of humans, by searching for and understanding relevant information published on the web or acquired via services. Such information could potentially be made available by multiple sources, using alternative ontologies, often with different provenance. Autonomous agents rely on communication languages and protocols to exchange data and coordinate their behaviour and thus collaborate. Early approaches based on speech acts (Austin, 1962), focused on message types or performatives (e.g., request, inform, and promise) based on a folk categorisation of the intended meaning of the communication. This evolved through the DARPA funded Knowledge Sharing Effort (KSE) resulting in a communication language, the Knowledge Query Manipulation Language (KQML), defining the mechanism by which agents communicated; and an ontology language, the Knowledge Interchange Format (KIF), describing the knowledge that the performative referred to (Finin et al., 1994). Although agents could perform services on behalf of their peers, discovered through capability registries (Decker et al., 1997), service invocation occurred as a by-product of requesting information. This contrasts with the notion of web services and things, which use web-based communication protocols, whereby the invocation of services could be requested explicitly (in a similar manner to calling methods or functions within a programming language) by providing the relevant input parameters, as data or knowledge fragments.

The prominent view from a Semantic Web perspective is that multiagent systems operate on the Web through the provision of services, using HTTP as the de facto standard transport protocol. Additionally, the Semantic Web community have developed standards, protocols, vocabularies, ontologies, and knowledge representation formalisms to facilitate the integration of machine-processible data from diverse sources at scale, using the existing web infrastructure. As such, the two communities diverged due to different priorities, though there is increasing recognition (Ciortea et al., 2019) that the Web is a natural application architecture for MAS and can support different types of interactions between agents and resources.

From a knowledge representation perspective, standards such as RDFS (Brickley and Guha, 2014) and OWL (Group, 2012) facilitate the representation of complex knowledge about agents, services, things and their relationship in an explicit and processable way. An example is the Provenance ontology (PROV-O), a data model for workflows expressed using agents, their actions, and other assets.666 Additionally, reasoning engines have been developed that are capable of reasoning over OWL ontologies, albeit often with some restrictions (cf., Pellet777, HermiT888, FACT++999, Racer101010 moeller/racer/, and RDFox111111 However, the use of ontologically grounded annotations for services within agent communication pre-dates the Semantic Web (Huhns and Singh, 1997; Fensel and Motta, 2001), and in some cases the Web itself (Huhns et al., 1993). Semantic Web service research exploited both F-Logic (Kifer et al., 1995) as used by WSMO (Roman et al., 2005), and DAML-S (Ankolekar et al., 2002) (based on the DARPA Agent Markup Language) which evolved into OWL-S (Martin et al., 2005). Other approaches to support service utilisation were developed using OWL, e.g., the OWL ontology for protocols, OWL-P (Desai et al., 2006), or using federated service discovery mechanisms such as the semantically annotated version of UDDI (Paolucci et al., 2002). These frameworks and ontologies were key in facilitating the discovery and use of services by autonomous agents, and provided an alternative communication paradigm built on web-based infrastructure. In addition, from the knowledge perspective, bespoke protocols were developed to support the decentralised management and exchange of knowledge and information amongst networks of agents or peers (Staab and Stuckenschmidt, 2006).

Other efforts include the provision of infrastructures for supporting the cleaning and validation of the data published on Linked Open Data Platforms; e.g., LOD Laundromat (Beek et al., 2014)121212 and OOPS (Poveda-Villalón et al., 2014).131313 Such techniques help detect errors in the data exchanged between agents and things. The SPARQL (Harris and Seaborne, 2013) query language facilitates federated querying over distributed data sources accessible via the web, whereas the Linked Data Platform (Speicher et al., 2015) can be used to manipulate RDF data via HTTP operations. Approaches have also been proposed to enrich SPARQL with qualitative and quantitative preferences (Gueroussova et al., 2013; Patel-Schneider et al., 2018) to select query results that satisfy user-defined criteria.

In recent years the Semantic Web community has broadened its focus beyond knowledge representation, reasoning, and querying to include knowledge extraction, discovery, search, and retrieval. However, many of the proposed tools and techniques have yet to be used extensively within MAS or by the MAS community. A recent survey (Kirrane, 2021) identified several open research challenges and opportunities in relation to the suitability of existing proposals for autonomous agent use cases, the combination of symbolic and sub-symbolic AI techniques for enhancing agent learning, and the development of tools and techniques for validation and verification.

3.3. Agents and the Web of Things

The Web of Things (WoT) (Kovatsch et al., 2020) refers to the Internet of Things (IoT) with an application of Web standards and technologies for improving interoperability of IoT devices and infrastructure. Things are resources that can be acted upon or queried via APIs (e.g., WoT scripting API (Kis et al., 2020)); autonomous goal-driven agents141414Here, we refer to agents in the sense of multiagent systems as discussed in Section 3.1. thus can make use of a WoT environment via WoT technologies and become part of the WoT ecosystem. Indeed, bringing agents to the Web requires more than simply exploiting Web protocols (such as HTTP (Fielding and Reschke, 2014)) and data formats (e.g., XML (Bray et al., 2008), RDF (Cyganiak et al., 2014)). The communication infrastructure used by agents should comply with an architectural style based on well-defined principles, such as Representational State Transfer (REST) (Fielding, 2000) as instantiated in the Architecture of the World Wide Web (Jacobs and Walsh, 2004).151515Note that the specification of the Web architecture defines the concept of Web agents as “a person or a piece of software acting on the information space on behalf of a person, entity, or process”. Furthermore, for things to be used without human intervention, they must be formally described. To this end, the W3C published the Thing Description (Kaebisch et al., 2020) standard, which specifies how a JSON-LD representation of thing affordances (i.e., properties or actions) via Web APIs can be provided. In addition, the WoT Discovery (Cimmino et al., 2020) standard provides a mechanism for the automatic discovery of thing descriptions (thus obviating the need to hard-code the location of such descriptions beforehand). These standards support improved heterogeneity by decoupling agents from thing implementation details.

The WoT activity highlights the importance of metadata with clear semantics, and made their standards, especially thing descriptions, compatible with RDF and Semantic Web technologies. In fact, even before a standardisation effort for the WoT started, multiple initiatives suggested the use of the Semantic Web to improve IoT systems (Rhayem et al., 2020). More precisely, in REST style hypermedia systems such as the WoT, things and agents are resources that interact by producing and consuming hypermedia about their state and the artefacts surrounding them (Ciortea et al., 2019). All resources are identified through IRIs161616Internationalised Resource Identifiers (Dürst and Suignard, 2005) to support global referencing, irrespective of contextual information. Therefore, resources can be represented through semantic descriptions that are expressed in a uniform data exchange format such as RDF using terms from some standardised and interlinked vocabulary expressed in OWL (Group, 2012). This standardised knowledge model hides the specifics of the implementation and facilitates interconnected resources that can be queried by exposing SPARQL endpoints. Of particular interest to WoT environments are the vocabularies that describe sensors and actuators (SOSA/SSN (Haller et al., 2017)), provenance (PROV (Lebo et al., 2013)), and temporal entities (OWL-Time (Cox and Little, 2017)).

The WoT provides a natural substrate for multiagent systems based on the vision that systems of interconnected things should be open and easily reconfigurable, and therefore such systems should comprise autonomous and collaborative components. This notion was supported by Singh and Chopra (2017) who argue that IoT systems need the kind of decentralised intelligence that MAS provides. Likewise, Ciortea et al. (2019) recommend integrating the Web and MAS to leverage the proven benefits of hypermedia systems for MAS. Importantly, these papers emphasise governance as a major challenge.

The technologies that emerge from the WoT community are often industry-oriented and paralleled by standardisation efforts. A recent example is the abstract WoT architecture design document (Kovatsch et al., 2020), supported by the Thing Description (Kaebisch et al., 2020) and the WoT Scripting API (Kis et al., 2020) specifications, for which a reference implementation is provided.171717 Although these technologies are more mature than MAOP technologies from an engineering perspective, and have a clear path to industry adoption, they lack the rich abstractions related to agents and autonomy that MAOP technologies provide. For example, the notion of a servient, as introduced in the WoT architecture design document can be considered an evolutionary step from a stricter server-client separation; a notion that is considered simplistic within the MAS community. Recent approaches have sought to form a bridge between the MAOP and WoT technology ecosystems (Ciortea et al., 2018b, a); however, this line of research is young and the corresponding technologies are nascent.

3.4. Norms, Policies and Preferences

Norms, policies, and preferences can help govern autonomous agent behaviour. The term norm has several meanings in natural language and is used widely in economics and social science. In MAS, the term “norm” typically expresses a deontic concept (e.g., a prohibition, permission, obligation, or dispensation). A coherent set of norms, i.e., created and evaluated as a unit, is referred to as an institution (Noriega, 1997). The same understanding of norms is found in the Semantic Web literature, where there is also a body of work focusing on policy specification and enforcement. Here, policy is an overarching term used to refer to a variety of system constraints, whereas the term preferences is primarily used in connection with privacy and personal data protection.

The study of norms is a long-running and active line of research within the MAS community, as evidenced by numerous Dagstuhl seminars (Dastani et al., 2018; Andrighetto et al., 2012), and a handbook on the topic (Chopra et al., 2018). Normative MAS (Boella et al., 2006) are realised and characterised in multiple ways, including those based on: (1) the agents reasoning capabilities; (2) whether norms are implicit or explicit; and (3) whether or not the architecture includes monitoring and enforcement mechanisms.

Agent capabilities vis-à-vis norms typically fall into three categories: (i) norm unaware, whereby agents may be regimented by external agencies to enforce norm compliance (Arcos et al., 2005); (ii) norm-aware, where agents may choose whether or not to comply with norms, depending on the alignment of their goals with those norms, the penalties for non-compliance, and the likelihood of enforcement (Shams et al., 2020); and (iii) value aware, whereby agents, in addition to being norm-aware, are able to participate in norm creation and norm revision, by reasoning about the values supported (or not) by particular norms (Cranefield et al., 2017). Thus, compliance in normative systems depends on how individual agents reason and adapt to norms at both design and run time (van Riemsdijk et al., 2013; Telang et al., 2019; Lee et al., 2014). Implicit norms that reside within the agents themselves are expressed through agent behaviour, but are not otherwise externally discernible, whereas explicit or referenceable norms may have an abstract representation involving variables and a grounded (detached) representation in an entity such as a contract (Singh, 2013), institution (Noriega, 1997; Esteva et al., 2002; d’Inverno et al., 2012), or organisation (Udupi and Singh, 2006; Boissier et al., 2020). Agents that are norm or value aware should be able to: (i) recognise norms; (ii) decide whether they want to follow them; and (iii) adapt their behaviour according to the norms, if they decide to do so. Such agents may additionally be able to engage in norm revision processes. Norms, and more broadly conventions or social norms (Lewis, 1969), are established in an agent society in one of two ways, namely top-down and bottom-up (Wooldridge, 2001; Morris-Martin et al., 2019).

In top-down systems, norms are identified as part of the MAS design process and are either: hard-coded into the agents’ behaviour (implicit representation), eschewing any form of normative reasoning and narrowing the scope for behavioural adaptation; or are prescriptive and explicitly represented, and thus external to the agents, typically represented in the form of abstract regulations (for example, ungrounded terms over variables) that, as a result of agent actions, become detached (for example, grounded terms over literals). The n-BDI variant (Criado et al., 2010) is a BDI-based agent architecture that allows for the internalisation of norms where the design suggests an agent-internal process that synthesise norm-style rules based on observed behaviour, whereas N-Jason (Lee et al., 2014) agents perceive institutional facts, which they internalise as beliefs and hence incorporate in their reasoning. Norms designed offline, however thoughtfully crafted for the long-term, are at risk of losing relevance in open, always-on, environments such as the Web, because it is not possible to anticipate all eventualities at design time. Furthermore, drift in the agent demographic or in systems goals, are likely to make norm revision essential over any sufficiently long system lifetime. With explicit norms, any norm change will affect the entire population. Such changes can be effected through a human-in-the-loop approach, where human designers revise the norms and then switch the system over at some suitable point; such as through a shutdown/reboot sequence, or the use of norm-aware planning (Shams et al., 2020). In the latter case, an agent must manage a plan sequence that although initially compliant, may cease to be part of the way through the plan due to the change in norms. Such an agent must also be able to check that its learned way of achieving a goal is compliant with the new norms, perhaps by means of some oracle (Padget et al., 2018), or by being able to acquire a fresh plan that is compliant.

In bottom-up

systems, an individual agent decides whether or not to adopt a norm: with implicit norms, it may seek advice from others or apply indirect reinforcement learning over its observations, as a basis for prediction, possibly in combination with a

strategy update function (Wooldridge, 2001). In such systems, norms are deemed to have emerged once they have been adopted by a sufficiently large fraction of the population; this is typically 90% in most of the literature, and 100% in some cases (which is hard to achieve), or assumes a simple majority, which can risk oscillatory outcomes. However, convergence (this term appears to be used interchangeably with emergence in the literature (Morris-Martin et al., 2021)) is a function of the capabilities of the agents. Emergence with explicit norms depends on agent reasoning capabilities. An agent might inform the regulator that it wants to take a particular action in a particular state (without sanction) – the agent knows what it wants but not how to get it—as a request to change the norms without having to reason about norm representation. A more difficult approach is that an agent might propose a new (abstract) norm – the agent knows how to define a new norm to get what it wants (Haynes et al., 2017; Morris-Martin et al., 2021). As above, changes have to be actioned, which could be as outlined previously, although pluralist approaches are possible, as put forward by Ostrom (1990), or by using one of the many voting mechanisms. The challenge for an agent then becomes how to decide which way to vote, which depends on their reasoning capabilities: are they able to evaluate the consequences of the norm change; and are they selfish (i.e., vote “yes” if the change is individually beneficial, e.g., increases their utility) or altruistic? (i.e., vote “yes” if the change is collectively beneficial). More sophisticated still would be the use of argumentation to determine if the revision is consistent with the population’s values (Shams et al., 2020; Straßer and Arieli, 2014).

In the early days, Semantic Web researchers proposed general policy languages, such as KAoS (Bradshaw, 1997), Rei (Kagal et al., 2003) and Protune (Bonatti and Olmedilla, 2007), which cater for a variety of different constraints (access control, privacy preferences, regulatory requirements, etc.). A prominent early attempt to provide a semantic model of polices as soft constraints for agents was OWL Polar (Sensoy et al., 2012), an OWL DL explicit policy representation language. OWL Polar aims to fulfil the essential requirements of policy representation, reasoning, and analysis, where policies are system-level principles of ideal activity that are binding upon the components of that system, and thus are used to regulate the behaviour of agents (Sensoy et al., 2012). Over the years the Semantic Web community have also proposed policy languages that are tailored to better cater for access control, privacy preferences, licensing, and regulatory governance requirements, including detailed surveys, for example, of the various policy languages, and the different access control enforcement strategies for RDF (Kirrane et al., 2017). From a privacy perspective, the Platform for Privacy Preferences Project (P3P) (Cranor et al., 2002) specification, deemed obsolete in 2018, aimed to allow websites to express their privacy preferences in a machine readable format that could be interpreted by agents that could automate decision making on behalf of humans. The P3P initiative, despite having failed, inspired subsequent work on representing and reasoning over privacy preferences, such as using OWL (Garcia and de Toledo, 2008), catering to more expressive privacy preferences (Kolari et al., 2005), and representing consent for personal data processing (Bonatti et al., 2020).

Many existing proposals rely on WebID (Sambra and Corlosquet, 2015), a community-driven specification that offers an identification mechanism making use of Semantic Web technologies to provide password-less authentication. An extension of WebID (specifically WebID-OIDC that relies on OpenID Connect181818 is used in the Solid project. Solid191919 is an ongoing initiative, lead by Tim Berners-Lee, aimed at deploying a distributed Linked Data infrastructure for governing one’s personal data, which is built on top of Linked Data Platforms. Additionally, there has been work on usage control in the form of licensing (Cabrio et al., 2014; Villata and Gandon, 2012; Governatori et al., 2013a, b, 2014), and more recently, policy languages have been used as a means to represent regulatory constraints (Palmirani et al., 2011; De Vos et al., 2019). The Open Digital Rights Language (Ianella and Villata, 2018; Fornara and Colombetti, 2019), although primarily designed for licensing, has been extended to cater for: access policies (Steyskal and Polleres, 2014); requests, data offers and agreements (Steyskal and Kirrane, 2015); and regulatory policies (De Vos et al., 2019). Usage control, however, often proves challenging for organisations and users, and any constraints imposed on the use of data need to ensure that policies are applied consistently across organisations and that there are robust propagation mechanisms preventing policies from becoming invalid (Daga et al., 2015, 2016). The notion of FAIR ICT Agents (Kirrane and Decker, 2018) is based on FAIR (Findable, Accessible, Interoperable and Reusable) principles (Wilkinson et al., 2016), where ICT denotes interactive intelligent agents that are constrained via goals, preferences, norms and usage restrictions. Thus far, the WoT standards offer only limited support for norms, policies and preferences, which are currently described in guidelines targeted at human developers rather than as declarative, machine-readable statements usable by agents (Reshetova and McCool, 2019).

Although research on norm-aware agents has made reasonable progress to date, much remains to be done to elevate human oversight to align with the three categories (High-Level Expert Group on Artificial Intelligence (AI HLEG), 2019): human-in-the-loop, where there may be human intervention in each decision cycle; human-on-the-loop, where there is human intervention in the design cycle and operation monitoring; and human-in-command, where there is human oversight of the overall system, including the means to decide when and how to engage the AI system. The motivated scenario presented herein draws on human-on-the-loop and human-in-command, and indeed it is these levels of abstraction that inspire the governance framework introduced in Section 4, since those are the system characteristics we aim to facilitate.

4. Conceptual Framework

The overarching goal of this section is to identify governance entities, their relations and their purpose, with no aim to be prescriptive in their instantiation. In doing so, we propose a blueprint for the governance of socio-technical systems that can be instantiated in a variety of ways, using a variety of concrete software components. Thus, this section aims to provide guidance for developers on how different parts of an agent governance system fit together and the functions that they contribute. Our objective here is to enable a range of solutions, fit for different purposes, realisable through available (rather than prescribed) software, but still coherent through the framework set out in the three layers shown in Figure 2.

Figure 2. Conceptual Framework for Governing Agents on the Web structured along three layers. Interactions (not represented here in this Figure) take place within each of the layer and between the layers.

4.1. Global View

In order to provide something actionable for designers and implementors, we ground our framework for the governance of autonomous agents on the Web through three layers that structure the various entities and abstractions needed for the development of socio-technical systems on the Web. Each layer is assigned concepts that are necessary for governance: norms, policies, and preferences (as illustrated in Figure 2). The way in which these different parts are realised, and how they interact is dependent on various design decisions. In setting out this framework, we draw on and organise existing work on norms, policies, and preferences (as described in Section 3) to cater for abstract requirements for the governance of socio-technical systems. This gives rise to the following three layers:

Reactive Things & Services Layer. This layer comprises non-autonomous entities in the environment. As seen in Section 3, such entities are key notions of the WoT architecture (Jacobs and Walsh, 2004) for which first-class abstractions are proposed for specifying and discovering them and other entities within the MAOP approach (e.g., artefacts in the JaCaMo meta-model (Boissier et al., 2020)). Such entities are perceived and acted upon by agents. We propose the use of policies for dealing with the governance of such non autonomous entities, following the same approach adopted by Semantic Web community. These policies state who can access them, and constraints on their usage (if any). Semantic Web technologies such as OWL-POLAR (Sensoy et al., 2012) can provide a means to implement, manage, and enforce policies that constrain access to things and services, and the affordances they provide.

Autonomous Agents (Human & Artificial) Layer. The agents layer is composed of entities that autonomously perceive and act upon their environment (i.e., things and services) and interact with the other entities. Agents are the main abstractions for specifying and managing autonomous behaviours. In contrast to the conventional model of programs on the Web as servers or clients, the WoT architecture introduces a servient that can both pro-actively access other things and services and reactively respond to requests from other things and services. In addition, servients can host one or several things. Whilst the Web architecture does not provide first-class abstractions for autonomy, it is possible to distinguish between “agentified” things that exhibit pro-active behaviours and reactive things by introducing custom properties into the W3C WoT Thing Description (Kampik et al., 2021). Agents have preferences that inform and constrain their actions with respect to things, web services, and other agents. Preferences control the local reasoning and decision-making undertaken by the agents, and can thus support governance. In traditional deliberation architectures for autonomous agents, preferences are specified (or emerge) as part of the often complex reasoning cycles. Hence, the management of these preferences given the presence of norms and policies can be challenging. Semantic Web approaches that consider preferences (e.g., SPARQL with preferences (Pivert et al., 2016)) can enable declarative preference management, especially when an agent’s preferences are to be considered.

Normative Organisations Layer. In MAOP, organisations are first-class abstractions (Boissier et al., 2020) that group agents and their governance (i.e., norms). Although the WoT architecture does not provide such abstractions, its security and privacy guidelines reflect similar notions to organisational norms. Whilst the previous two layers (discussed above) included governance concepts dedicated to the local governance of each entity (e.g., policies for thing, preferences for agent), this layer addresses the governance of autonomous entities participating in the system. This layer manages abstractions for the logical grouping of agents with a particular purpose, and the provision of legal, regulatory, and social norms that may possibly span multiple organisations. However, organisations are entirely virtual and passive (i.e., shaped by their members), thus it is up to these member agents to stipulate, comply with (or violate), enforce, and evolve organisational norms. Semantic Web technologies such at ODRL (Fornara and Colombetti, 2019) allow for the formalisation of norms for specific domains and purposes; hence, they can be integrated seamlessly with the more abstract MAOP abstractions for organisations and norms that are agnostic to these details.

From a MAS perspective, this framework is coherent with the JaCaMo meta-model (Boissier et al., 2020); from a WoT perspective, it is coherent with the WoT architecture (Jacobs and Walsh, 2004); and furthermore, it is coherent with the Semantic Web perspective, although with enhancements with respect to policies, preferences, and norms. It is worth noting that our conceptual framework provides software engineering abstractions. Analogously to the Web architecture, we do not recommend a one-to-one mapping of software abstractions to physical entities (devices). Considering Web architecture standards, the WoT Scripting API supports, for example, the instantiation of multiple things as part of one servient, which may represent a single physical machine.

4.2. The Layers

In this section, we return to each of the layers introduced in Section 4.1 and detail their composition and their governance.

4.2.1. Reactive Things & Services

As defined in Section 3.3, things are physical objects that are endowed with network capacities that allow one to make use of their functions in a digital environment. For example, they can be sensors that provide measurement data through the Internet, or actuators that can be triggered from a Web API. Therefore, in the WoT context, things resemble and are sometimes assimilated within web services. These web services are normally purely digital entities that simply exchange data via their input parameters and output results. In the WoT architecture, things may be autonomous, whereas, in contrast, our conceptual framework distinguishes between autonomous agents (which may be things in the WoT architecture) and reactive things.

When it comes to things and services, policies serve many purposes. Access control policies ensure that only authorised agents use specific things and services. Here, there is a need to provision both authentication and authorisation mechanisms, and policies may help resist security threats. Additionally, policies may govern the use of data that is produced by things and services; e.g., to ensure personal data protection or intellectual property rights.

From a policy governance perspective, it is useful to distinguish between enforcement and compliance of the autonomous agents acting on these things and services given their respective policies. Enforcement means that any violations are prevented, whereas compliance means there is a need for retrospective conformance checking.

4.2.2. Autonomous Agents (Human & Artificial)

In contrast to (reactive) things and services, agents are entities that pursue their own goals autonomously. They determine the necessary actions that should be executed on the things and services situated in the environment. In the MAS literature, several agent architectures that are based on the different properties exhibited by the agents have been proposed (Wooldridge and Jennings, 1995). They range from purely reactive (i.e., those that respond to stimuli without complex symbolic reasoning to reason about future actions) to deliberative ones (those that maintain a symbolic world model for reasoning about plans and decision making) (Calegari et al., 2021). A notable example of deliberative architectures is the BDI architecture (Rao and Georgeff, 1995), where agents are programmed using their mental attitudes such as beliefs, desires, wishes, etc. (Bordini et al., 2007) and that is one of the mainstream architectures for cognitive agents in MAS.202020A number of different MAOP frameworks that adopt a BDI architecture were discussed previously in Section 3.1. This contrasts with reactive architectures (such as the subsumption architecture (Brooks, 1991; Theodorou et al., 2017)) typically used by robotic systems, whereby behaviours define the actions a robot should perform as a consequence of some stimuli (e.g., from sensor data or direct communication). Many hybrid agent architectures (Wooldridge and Jennings, 1995; Bryson and Stein, 2001) combine elements of both reactive and deliberative ones, where prominence is often given to the reactive aspect over the deliberative aspect (such as obstacle avoidance versus goal deliberation). Our conceptual model focuses on governance and is agnostic with respect to any particular architecture, and thus cater for the heterogeneity of agents.

In addition to taking decisions on their own, agents may also coordinate with humans or with other agents to adjust and align their goals with the other agents’ goals and identify joint goals, and as such, they may communicate with other agents or human users by exchanging messages. We address the various means of interaction among agents in Section 4.3 below.

Each agent maintains a representation of its internal state that is built from the agent’s internal reasoning, from its perceptions of the environment, i.e., the observable state of the things and services deployed in the system, and from its interactions with other agents. Acting on behalf of human users (e.g., assistant agents) or abstract entities (e.g., service agents), agents manage preferences that guide their decision process. It is important to differentiate between agents developed by the application designer and those that enter the system at run-time. This differentiation emphasises the level of control the application designer has over the agent with respect to its internal state. It also justifies the proposition of two levels of governance within our conceptual model: preferences for local and individual control; and organisations for global and collective control. In our conceptual framework, preferences cover many dimensions, ranging from privacy preferences to moral values or ethical principles. Additionally, there can be either agreement or conflict between preferences and access control as defined in the previous layer, due to the fact that an agent may need to verify someone’s identity, and based on this determine what information to disclose. Being part of a MAS, the reasoning and decision mechanisms of the agents are enriched with mechanisms to reason over several factors including: norms; regulatory requirements coming from the organisation in which the agent participates; and over policies or access control rules attached to the resources, things, and services with which the agent interacts.

4.2.3. Normative Organisations

Organisations act as coordination mechanisms by which agents work together to achieve their joint goals. The design of agents within an organisation focuses mainly on the agents’ capabilities and constraints, as well as on organisational concepts such as roles (or functions, or positions), groups (or communities), tasks (or activities) and interaction protocols (or dialogue structures); therefore on what relates the structure of an organisation to the externally observable behaviour of its agents (Ferber et al., 2004). Organisations usually have a structure defined by: (i) groups, whereby agents are classed together and possibly organised hierarchically; and (ii) roles, whereby agents assume various duties. For example, agents can belong to multiple organisations, be part of various groups, assume different roles (possibly at the same time), and join or leave organisations at will. Organisations can be formed at design time or emerge due to interactions between agents at run-time.

The dynamics of the organisational structure, for example an agent changing its role or joining a group, is governed by rules that are formalised as the norms of the organisation. Norms define what communication is possible, allowed, or forbidden between agents. An organisation is a means to regulate agent behaviour, and such organisations may be governed by norms, including laws and regulations adopted from the social setting or jurisdiction and those legislated within the organisation. The organisation structure and its normative part are described in such a way that agents can autonomously take part in the organisation and regulate themselves automatically with the aim of achieving their (individual or collective) goals. However, a formal, explicit encoding of norms is necessary to facilitate automated compliance and conformance checking.

4.3. Interactions Within and Among the Layers

Within a MAS that is fully aligned with our layered conceptual framework, several interactions may take place within each of the layers and across them (i.e., both inter and intra-layer interactions). Similarly to the Web architecture, the conceptual framework is protocol-agnostic. Some protocols may be chosen based on the underlying things and services to be used; whereas other protocols would be custom to the desired agent-agent interactions; and some of these latter protocols may be designed whereas others evolve. We identify the following types of interactions in the conceptual framework:

Agent-to-agent interactions. Agents can interact with other agents directly, by exchanging messages or acting upon each other, or indirectly, by observing each other’s actions on the reactive things and services of their environment. Because agents are autonomous, the requests that one agent sends to another are handled at the discretion of the receiving agent. In comparison to the interaction with Web services, interaction with an agent may imply a higher likelihood that the response deviates in a complex and nuanced manner from the requested resource.
Agent-to-thing/Thing-to-agent interactions. Agents proactively interact with things and services by acting upon them, accessing their properties, and by listening to (perceiving) events that things and services emit.
Thing-to-thing interactions. While things and services are purely reactive, they may interact with other things or services as part of a reaction chain. In this context, existing standards that are part of the Web architecture can be applied for basic communication, but more expressive approaches may be required to manage norms, policies, and preferences, for example when a thing communicates on behalf of an agent across organisational boundaries.
Agent-to-organisation/Organisation-to-agent interactions. An agent’s preference depends on the norms of the organisations that the agent is a part of. However, because the agent is an autonomous entity, it may choose to not adopt an organisational norm. At the same time, the agent may attempt to change an organisation’s norm, for example by proposing a norm update that then requires approval by a majority of the organisation’s agents.
Thing-to-organisation/Organisation-to-thing interactions. In contrast to agents, things and services cannot directly affect organisations. Things and services can be implemented to dynamically adopt policies that reflect organisational norms, and the state of a thing or service can be considered by an organisation, but in both directions, the organisation is the leading system.
Organisation-to-organisation interactions. Several organisations may have (unidirectional or bidirectional) dependencies. For example, in a hierarchy of organisations, the norms of lower ranking organisations may depend on norms that are specified on a higher level in the hierarchy; still, a higher ranking organisation may have some norms that depend on the norms of multiple lower ranking organisations (consider dependencies between a federated state and its federal entities).

In the MAS community, interaction protocols are typically designed from a global perspective and aim to facilitate interaction and coordination between agents. A protocol specifies the permitted enactments; i.e., the possible sequences of message exchanges. Proposals for languages for interaction protocols include process algebra (Ferrando et al., 2019), Petri Nets (Reisig, 1985), and information protocols (Singh, 2011). Petri nets may then be mapped to models that are more accessible to human users, such as Business Process Model and Notation (BPMN) diagrams. In practice, protocol design and protocol discovery can go hand-in-hand: in particular, Petri Net-based protocols (processes) can be mined from IT system event logs (Van Der Aalst, 2012), which, for example, can be used for organisational compliance checking (Caron et al., 2013). Recently, agent system mining has been proposed as a novel process mining variant that focuses on the agents that participate in one or several (organisational) processes, i.e., on the micro-level instead of on the macro-level process view that an organisation imposes (Tour et al., 2021). In the service-oriented community, the notion of a choreography is similar to an interaction protocol (Barker et al., 2009), in that a choreography describes interactions between services from a global perspective.

Human-Agent interactions are typically modelled as conversations between the different agents, i.e., dialogues (Walton and Krabbe, 1995). A dialogue has a normative aspect: it is regulated by norms, and can establish new norms. In normative systems, dialogue protocols are specific notations for norms that specify the violation contexts. Utterances in a dialogue can be seen as moves in the underlying protocol that create obligations and permissions for the participating agents. Of particular interest are persuasive dialogues, where an agent can convince, suggest, or command. Agents can use persuasive dialogues to convince other agents to add new beliefs; to enter into some form of negotiation; or, in the case of the command, a new violation rule is introduced thus creating a new obligation (Boella et al., 2004).

Besides the protocols themselves and their logical organisation in process choreography, the Agent Communication Language (ACL) and the agreed vocabularies are crucial when it comes to the interaction and co-ordination of agents in a MAS (Section 3.2). The Web architecture specification lists properties, actions, and events as the central abstraction of its interaction model. In our conceptual framework, organisations, agents, things and services may expose properties and generate events, but only agents may execute actions. From a governance perspective, there is a need for policy, preferences, and norm-aware interaction protocols. For example, agents may need to authenticate themselves to other agents as well as to things and services, whereas collaborating agents may need to engage in preference elicitation and negotiation, and norms may need to be communicated and possibly agreed upon by agents that form part of an organisational structure.

5. Use Case Revisited

In this section, we demonstrate how a normative MAS that leverages web services, things, and Semantic Web technologies could be used to realise our motivating scenario (Section 2). We show how several example situations can be modelled using the proposed conceptual framework (Section 4) and highlight technologies that could be used to instantiate our governance framework.

Figure 3. Normative organisations, autonomous agents, and reactive things and services in the scenario.

5.1. The Global Setting

Agents encapsulate knowledge, goals, and preferences corresponding to the autonomous entities involved in the vaccination process. The resulting conceptual model is illustrated in Figure 3.

An assistant agent is in charge of managing personal data on behalf of a patient (e.g., the patient John). A physician agent is in charge of managing administrative tasks to act on behalf of the physician (e.g., Jane). Other types of agents access the things and services (i.e., a vaccine guard agent controls access to the freezer), and to manage the vaccination process by collecting patients’ data and checking their eligibility (i.e., scheduling system agents). It is worth noting that, contrary to the other agents, the first two kinds of agents (i.e., assistant agents and physician agents) may not be under complete control of the stakeholders who develop and own the application.

We introduce a vaccination centre organisation to delimit the vaccination application and to provide scope for the adherence to regulations and behaviours for both artificial and human agents that are part of this structure. To this end, the organisation specifies roles and norms, whereby the roles are used to structure agent responsibilities, and the norms (i.e., duties, rights, and interdictions) regulate the vaccination application. Agents with a given role are expected to fulfil the corresponding norms. The vaccination centre is, in turn, part of the health service organisation of a particular state, in which clinic organisations complement the normative framework provided by state and vaccine centres. In addition to norms, the definition of the organisation may impose hard constraints on its composition that should be enforced by service policies. For example, by stating an upper limit on the number of agents that can adopt specific roles, the vaccination application may consequently limit the number of patients or physicians that may enter the organisation.

5.2. Illustrative Use of the Conceptual Framework with our Motivating Scenario

The following paragraphs describe the use of the conceptual model in situations derived from the motivating scenario. For narrative convenience, we use the terms obligation, permission, and authority in an informal sense.

5.2.1. Initialising the Vaccination Application

At the launch of the application, the vaccination centre organisation is created, by endowing the agents that support the business processes within the vaccination centre organisation with the roles necessary to fulfil their goals. The definition of the organisation (e.g., the roles and distribution of norms on the roles) is published as a web resource in a machine readable and understandable format, accessible to any agent wishing to become a member of that organisation. The current state of the organisation (i.e., which agent is assigned the various roles) is published and updated as necessary, over the entire lifetime of the organisation. The freezer agent adopts the guard role, which results in it being assigned the duty of managing access to the inventory of COVID-19 vaccine doses stored in the freezer. It obtains the permission to use the robotic arm to retrieve a vaccine dose when asked, and to deliver it to the staff. The manager agent is assigned the organiser role, and consequently inherits the obligation to compile lists of eligible patients based on the patient data and the vaccination eligibility policy. The data agent adopts the collector role and obtains the authority to collect personal information about each patient requesting a vaccination appointment; it also has the obligation to verify the patient’s eligibility for receiving the vaccine as well as the obligation to solicit patients through dissemination channels when vaccine doses and scheduling slots are available.

5.2.2. Joining the Vaccination Centre Organisation

When a patient obtains the credentials to access and use the vaccination application, the assistant agent acting on behalf of the patient is provided access to the web resource describing the organisation. After reasoning over its obligations and authorities, as imposed by the vaccination centre organisation, the agent decides to adopt the role. The assistant agent subsequently acquires the obligation to provide access to the patient medical data. This role may create internal conflicts between preferences provided by the patient and the obligations assumed when the agent took on the patient assistant role. After accessing and reasoning about the description of the vaccination centre organisation, the physician agent discovers that it has the obligation to coordinate with agents that are assigned to other roles. To assume the medical practitioner role, the physician agent must authenticate itself; upon adopting the role, it captures the associated permissions, obligations, and authorisation for further decision making. The same process of role adoption applies to other agents.

5.2.3. Assessing Patient Eligibility

While assigned to the organiser role, the manager agent takes into account its preferences in defining the patient information collection policy, and sends it to the agent with the collector role (as stated by the organisation definition). Fulfilling its obligation, the agent checks the eligibility of all arriving patients so that each dose is only administered to an eligible patient and that doses are administered before their expiry date.212121In a practical scenario, we would not expect 100% compliance with this constraint, but rather that the number of excess or wasted doses (relative to administered doses) does not exceed a specified threshold. To fulfil its goals, the agent therefore requests that agents adopting the new patient assistant role share the necessary patient personal data. It is worth noting that agents with the collector role need to consider the obligations stated by the organisation as soft constraints, and identify contexts in which these constraints may be relaxed. Further complications may arise if any of the agents attempt to negotiate relaxations of these obligations, either in anticipation of, or after a (perceived or factual) violation. For example, an assistant agent can negotiate an exception for a potential obligation violation by using computational models and algorithms of formal argumentation where the assistant agent believes that the data it has for its patient satisfies the eligibility criteria. On behalf of the organisation, the agent that has adopted the organiser role is in charge of the definition of the eligibility policy, and consequently may interact with the agent in charge of the data collection by granting or denying the request for an exception, or even by updating the organisation’s norms in order to accept the request.

5.2.4. Administering the Vaccine

When administering the vaccine, the agent with the physician role must respect the priority order for vaccine administration as defined by the agent in charge of the collection; for example, the elderly and vulnerable population must be vaccinated first, unless respecting the priority order implies wasting the dose. Importantly, the physician must not violate this priority order by, for example, preferentially vaccinating friends or relatives. In some cases, the physician agent must choose between either administering a vaccine dose despite the eligibility status being uncertain, or allowing the dose to expire. For example, existing regulations indicate that administering a vaccine dose to close relatives is impermissible, but the agent may conclude that in accordance with practitioner norms (such as the Hippocratic Oath), it is preferable if vaccines are not wasted. In such cases, the agent may need to prepare a defence strategy to avoid sanctioning, for example, via argumentation approaches (Baroni et al., 2018; Bench-Capon et al., 2009). These issues merely relate to the permissions needed for the obligations implied by opening the fridge. Additional challenges arise when considering the complete socio-technical system, including electronic health record access (Kafali et al., 2020) and supply chain integration (Singh et al., 2009).

6. Challenges and Opportunities

We now present research challenges (i.e., technical limitations of existing proposals) and opportunities (i.e., open research questions) related to normative agents on the Web. The four horizontal challenges that characterise the contributions of norm-based multiagent systems for the Web are described below, in addition to two orthogonal challenges that need to be tackled in order to address the horizontal issues (and illustrated in Figure 4). For each area, we label the challenges in the context of limitations of the state of the art and subsequently identify future research opportunities. Figure 4 also indicates the practical maturity of each challenge, from nascent (blue sky challenges with basic research potential, using a white background) via developing (basic research with immediate practical potential, using a white-grey gradient background) to practical (challenges that can be addressed primarily from an engineering perspective using a grey background).

6.1. Relating Norms and Interaction Protocols


A broad challenge in engineering normative MAS is that we need a way to operationalise norms in the sense of giving them a computational interpretation. Interaction protocols characterise interactions based on message order and occurrence – that is, in operational terms. However, it is nontrivial to produce protocols that are as flexible as necessary, yet enactable in a decentralised manner, while at the same time being verifiably correct. Although the W3C provides Web-based standards for retrieving and querying machine-readable data, these standards do not cater for usage constraints, such as access policies, intellectual property rights, and privacy preferences. In our scenario, an agent may, for example, want to decide with whom and in which context it shares its vaccination status. Existing work on interaction protocols (FIPA, 2003) largely focuses on request-response interactions and imposes restrictions on computation for scenarios involving the interaction of three or more parties (Ferrando et al., 2019; Chopra et al., 2020). In particular, traditional approaches entwine control flow details into the protocol, thereby making it difficult to separate them from the content, for which a declarative meaning can be specified. Prior work on specifying protocols based on norms (commitments) (Chopra and Singh, 2004; Mallya and Singh, 2007) was hindered by the lack of declarative specification of the constraints on messages. More recent approaches describe causality and integrity constraints on messages declaratively (Singh, 2011); such protocols (whilst sufficiently flexible to support all enactments of the stated norms) can grow quite large (Singh and Chopra, 2020), but emerging verification approaches aim to tackle this challenge (Singh and Christie V, 2021).


When it comes to operationalising norms, from a service-provisioning perspective, there is a need to develop policy-aware querying and data retrieval protocols; whereas from an agent interaction perspective, norms should be mapped both to the agent platform and the environment. This raises several important questions, including: (i) How can we design norm-aware dynamic interaction protocols? (ii) How can existing querying and data retrieval protocols be extended, such that they are policy aware? (iii) What new languages are needed to facilitate norm governance? (iv) How do we model and reason with respect to norm changes and temporal validity?

When operating in deployed systems modelled according to the conceptual framework described in Section 4, these interaction protocols also need to take into account that the autonomous agent layer includes human agents which might have conflicting requirements, and therefore strategies might need to be employed in order to resolve conflicts. This raises a number of additional questions that need to be addressed, including: (i) How do we ensure protocol compliance by human agents? (ii) How do we model protocols that implement persuasion? (iii) What mechanisms do we use to resolve conflicting requirements?

Figure 4. Overview: Research challenges for normative MAS and the Web

6.2. Distributed Normative MAS and Open Organisations on the Web


Organisations, institutions, and contracts are useful abstractions to structure norms and make them accessible to agents. Although agents have the choice of joining such structures, they may be subject to conditions that regulate their admission (and exit), as well as there being an expectation to comply with the organisation’s norms. Due to the scale of the Web, numerous permanent, ephemeral, or evolving structures may exist. Consequently, an agent needs to be able to discover and reason about such organisations and the corresponding norms. In the vaccination scenario, for example, an agent may need to be able to discover organisations that model the healthcare systems of other jurisdictions that potential patients may need to refer to, when claiming that they are eligible to receive a specific type of vaccine dose. Ontologies facilitate the discovery of services (Paolucci et al., 2002), and their use as a means to represent organisations is promising. A major challenge related to the distributed management of such structures (Boissier et al., 2020) is to monitor and enforce norm compliance, and to instantiate organisations, agents, or complete multiagent (sub-)systems at run-time on the Web, which is an emerging line of research in the MAS community (Amaral et al., 2020; Amaral and Hübner, 2020; Aldewereld et al., 2010). Another challenge is that agents require abstractions and mechanisms to build and adapt organisations on the fly (Chopra and Singh, 2016). Additionally, an agent may participate in multiple structures that operate at different timescales and scopes, and hence accommodating their diversity is nontrivial.


Addressing the above challenges requires answers to the following research questions, in the context of Web-based and WoT-based technology ecosystems: (i) How can agents discover organisational norms on the Web? (ii) How can norm compliance be monitored and enforced in dynamic scenarios, in which agents, organisations, or entire (sub-)MAS are instantiated at run-time? (iii) How can autonomous agents create and change organisations on-the-fly? (iv) How can normative organisations accommodate agents that participate in multiple organisations, with potentially inconsistent norms, and partial semantic interoperability? (v) How can compliance checking and enforcement approaches that are prevalent in the information systems literature be adapted and applied to normative MAS on the Web?

6.3. Internal State of an Agent and Norms


An agent should be able reason about norms, taking into consideration its internal state (e.g., its beliefs, goals, and intentions), and explain its normative reasoning to others. This is, for example, important for the administering the vaccine scenario (Section 5.2.4), when a decision is to be made about whether to administer a vaccine to a patient whose eligibility status is uncertain, using for example qualitative methods (Parsons, 2001), argumentation (Baroni et al., 2018)

or sub-symbolic techniques such as classification or Bayesian networks. Several languages express and support automated reasoning about agent internals, such as beliefs, desires or goals, and intentions. However, challenges exist when it comes to reasoning not only with respect to goals, but privacy preferences, regulatory constraints, and norms. The problems here not only relate to knowledge representation – how to represent all these aspects so they are accessible to an agent – but also to the impact such enrichment may have on deliberation performance. The most popular (symbolic) agent architecture is practical reasoning by means of beliefs, desires and intention (BDI), but this offers no guarantees about how long the deliberation cycle may take. Indeed this falls to the developer, in as much as they can endeavour to keep rules relatively simple whilst limiting the number of overlapping conditions, but the actual response time is out of their hands. Additionally, there is the more significant problem that arises due to the fact that agent architecture research has primarily focused on agent internals. Irrespective of whether they are comprised of symbolic or sub-symbolic aspects (such as reasoning, reinforcement or probabilistic learning), such architectures are not normally conceived or designed for interfacing with non-agent technological frameworks and their underlying abstractions (for example as provided by the W3C Web architecture and related standards).


From a norms perspective, important open questions in specification and enforcement include: (i) How can we ensure consistent representation of and adherence to norms? (ii) How should a governance architecture be designed in which rational agents are incentivised to comply with norms? (iii) Can a norm violation be excused, based on explanation or argumentation? (iv) Could transparency facilitate the persuasiveness of the explanation or argument? (v) How do we ensure that an agent is aware of the implications of violating a norm? (vi) How do we cater for agents that are not rational (as such agents may not be designed for criteria such as rationality)? (vii) How should performance limitations affect normative decision-making in practical reasoning architectures? (viii) How can normative reasoning be implemented in real-world agent architectures (which may only be agent-oriented in the broader sense)? (ix) How can agent architectures be better integrated with web technologies and standards?

6.4. Governing Norm Emergence


Approaches for the governance of norm emergence are dependent on the capabilities of the agents in a MAS, bearing in mind that population properties may not be homogeneous. In our example scenario, the governance of norm emergence is, for example, important to facilitate vaccinations (i.e., the belief that getting vaccinated is, while typically not mandatory, good for one’s health and more broadly for the public health at large), and to balance “hard” rules and “soft” recommendations to decrease the spread of COVID-19. The challenges here include modelling and managing the spread of beliefs and counter-beliefs, the potential resolution of contrary positions through argumentation, and how to make hard and soft policies accessible to different agent architectures with different reasoning capabilities. We differentiate between a decentralised approach to norm emergence with implicit norms, where the norms emerge through the interactions of agents – (Ajmeri et al., 2018) is one example of such a scheme – and various centralised approaches to the governance of norm emergence (Morris-Martin et al., 2019)

, which latter we classify by adapting the oversight terminology put forward by the 

High-Level Expert Group on Artificial Intelligence (AI HLEG) (2019, §B.II.1.1): (i) an external agency observes the behaviour of the population to identify patterns of behaviour and revise norms imposed by that agency to optimise for system goals (external agent/human on-the-loop) – for example Morales et al. (2015) look at individual norms in isolation – while the general case of revising a consistent body of norms remains open; (ii) agents propose norm revisions to an external agency, which then implements them subject to an assessment of how those revisions contribute towards system goals (external agent/human in-command), which also remains open; and (iii) agents propose norm revisions and system participants, which may include humans, use an internal decision-making mechanism to establish which changes will be implemented (internal agents/humans in-the-loop). The uHelp system illustrates some preliminary steps in this direction (Osman et al., 2021), but relegates software agents to a supporting role. The human-in-the loop, human-in-command, human-on-the-loop approaches are closely related to the different strategies that regulate persuasive dialogues (Boella et al., 2004), where commands introduce new obligations, whilst suggestions introduce new beliefs. However, open challenges relate to devising persuasion strategies and the corresponding obligations.


In order to govern norm emergence in Web-based MAS and their socio-technical contexts, one needs to answer – for example – the following questions:

(i) How can the emergence of norms in conjunction with governance decisions be monitored and managed in Web-based MAS? (ii) What roles do human-on-the-loop, human-in-the-loop, and human-in-command approaches play in the context of the preceding question, and what are the engineering implications of these different human interaction approaches? (iii) What is necessary to maintain alignment of the (evolving) value preferences of participants with the norms that govern them: when does one norm change become many changes? (iv) Which collective decision-making mechanisms are best suited for all agent and for mixed human-agent systems? (v) What is the appropriate capacity for agent reasoning about (self-)governance? Is wanting to do something that is prohibited, or not wanting to do something that is obliged a sufficient statement of intention? (vi) How can an agent and/or a human evaluate the consequences of norm revisions? Will they create a fresh problem while resolving the current one? (vii) How can oscillatory norm change be prevented? An agent-dominated system could potentially change faster than a human-in-command can evaluate the changes.

6.5. Heterogeneous and Inconsistent Norms and Beliefs


In heterogeneous information systems, we cannot reasonably assume that norms and policies are globally accepted and thus agents may hold inconsistent beliefs about them. For example, in our vaccination scenario, vaccine administration policies, eligibility requirements, and IT system landscapes may differ between two federated states A and B. However, a patient who moves permanently from A to B should ideally be able to receive the first vaccine dose in state A, and a dose of a matching or complementary type in state B after a reasonable time interval. For aligning norms and policies of sub-entities, reaching (partial) agreements in the face of conflicting beliefs regarding norms and policies is an important challenge that needs to be tackled to enable normative distributed MAS on the Web; using, for example, long-running lines of research on agreement technologies (Ossowski, 2012) and formal argumentation (Baroni et al., 2018). Currently, the body of research on belief revision and argumentation-based reasoning is, however, poorly integrated with practical engineering perspectives; standardisation efforts like the specification of an argument interchange format (Rahwan and Reed, 2009) exist, but have not found substantial adoption.


There are several open questions when it comes to enabling reasoning and decision-making in the face of inconsistent norms and beliefs of agents on the Web, including: (i) To what extent is there a practical need for engineering abstractions that treat conflict and inconsistency in the context of a normative Web (of Things)? (ii) What systematic approaches to drawing inferences and making decisions in a Web (of Things) governance context can be designed, implemented, and standardised as software engineering abstractions? (iii) How can existing research on belief revision and argumentation-based reasoning be made more accessible both from an engineering and a standardisation perspective?

6.6. Technological Integration


In order to facilitate the practical applicability of research on norms and policies for autonomous agents on the Web, it is crucial to build bridges across the technology ecosystems of the different communities. Section 3 provides an overview of the technology ecosystems that have emerged from the MAS, Semantic Web, and WoT communities. To summarise, in the WoT community, engineering-oriented work has been conducted in a highly practice-oriented manner, in close alignment with industry practitioners as well as standardisation bodies such as the W3C. An example of practice-oriented work can be observed though W3C IoT standards that feature an abstract architecture (Kovatsch et al., 2020) and an interface specification (W3C WoT Scripting API) (Kis et al., 2020), supported by a JavaScript reference implementation.222222 Research on engineering autonomous agents and MAS has primarily gained traction within the academic community (Mascardi et al., 2019), and standardisation attempts such as FIPA232323 have lacked significant adoption. Adjusting agent-oriented programming and software engineering approaches to better serve the Semantic Web and WoT communities is a way for the MAS community to move their engineering research closer to practice. This lets us conclude that while each of the communities has its own thriving technology ecosystem, a key challenge lies in integrating these ecosystems, which exhibit different degrees of practical maturity.


The above observations raise two questions: (i) How can the technology ecosystems of (normative) MAS and the Semantic Web be integrated with the WoT, and in particular with the W3C Web architecture? (ii) How can issues of practical maturity be mitigated (by the integration strategy)? With respect to (i), we argue that an integration strategy can employ a combination of two approaches across two dimensions that requires pragmatic trade-offs, considering the discrepancies between the technology ecosystems and their underlying conceptual abstractions.

Approach 1: Full-Fledged Framework Adoption. In order to facilitate implementations that build on research in the different communities, interfaces that integrate Semantic Web, MAOP, and WoT technologies can be devised that either re-implement their abstractions or integrate technology frameworks and specification languages (Ciortea et al., 2018b, a, 2019). A benefit of this approach is that it facilitates the adoption of powerful abstractions and technology ecosystems developed in these communities. A disadvantage, however, is that this approach can cause a high technological overhead.
Approach 2: Modular Abstraction Adoption. In order to facilitate implementations that build on technology stacks established in industry, minimally viable abstractions on norms and autonomous agents can be implemented as reusable modules in mainstream programming languages; or alternatively, specific features of complex technology platforms can be exposed as service-oriented interfaces. This strategy resembles a call to action as made in Logan’s Agent Programming Manifesto (Logan, 2018) and allows for deliberate trade-offs between conceptual richness and practical feasibility by avoiding the overhead (on conventional developers) of having to learn unfamiliar programming paradigms. For example, one may adopt JaCaMo’s capabilities for modelling organisations and artefacts via a Java-based technology stack, and defer adopting Jason (Bordini et al., 2007) since it involves a custom language for agent-oriented programming.

Broadly, the two approaches can be considered analogous to the integrated system (Approach 1) and best of breed (Approach 2) strategies for implementing large scale enterprise systems (Light et al., 2001). In an actual implementation scenario, these approaches represent the extremes of a scale with valuable trade-offs in between. We suggest that this trade-off is initially made from a conceptual perspective (Which programming abstractions are useful in a given scenario?) and followed using a technology perspective (Which technologies do I want to use to implement these abstractions?).

With respect to the second question, we argue that the strategy should prioritise mature technologies, and if necessary re-implement the requisite abstractions in technology stacks that are established in industry practice. Specifically, we might consider the WoT standards and technologies as a mature foundation on which to place Semantic Web and MAS technologies. An example of a synergy is in extending WoT servients to autonomous agents without necessarily committing to a BDI architecture for those agents.

7. Conclusion

This paper discusses the relevance of norms, policies, and preferences for governing complex socio-technical multiagent systems on the Web. The key challenge – the conceptual and technological integration of normative concepts with WoT abstractions and systematic evaluation of the practical usefulness of the integration results – is aligned with the general challenge for autonomous agents on the Web to transfer the rich theoretical achievements of the broader MAS community to the practical and engineering-oriented WoT community, and to facilitate real-world applications at scale. While the challenge of transferring research on normative agents and multiagent systems into engineering practice is well-known and generally acknowledged, this paper has taken the emergence of new Web standards, as well as the increased research interest in Web-based MAS, as a starting point to provide a new and broad perspective on it, with a focus on the Web and Web of Things Architecture standards. In this context, the paper proposes a conceptual framework that serves to define the role played by various norms, policies and preferences when it comes to complex socio-technical MAS on the Web, and demonstrated it via a simple but realistic scenario.

In addition, the paper provides a research roadmap outlining the technical and theoretical research challenges and opportunities to support complex socio-technical MAS governance on the Web. In particular, this roadmap calls for: (i) relating norms and interaction protocols; (ii) incorporating normative organisations and norm governance approaches into WoT architectures and standards; (iii) combining agent reasoning to relate policies, preferences, and norms; (iv) tackling the emergence of norms for flexible governance; (v) designing reasoning methods about norms in the face of inconsistency; and (vi) cautiously advancing Semantic Web and (normative) MAS tools and frameworks into practice via the WoT.


We thank the anonymous reviewers. Timotheus Kampik is supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Adnane Mansour, Olivier Boissier, and Antoine Zimmermann are supported by the French National Research Agency under grant ANR-19-CE23-00030 (project HyperAgents). Sabrina Kirrane is funded by the FWF Austrian Science Fund and the Internet Foundation Austria under the FWF Elise Richter and netidee SCIENCE programmes as project number V 759-N. Munindar P. Singh acknowledges support from the US National Science Foundation under grant IIS-1908374.


  • N. Ajmeri, H. Guo, P. K. Murukannaiah, and M. P. Singh (2018) Robust norm emergence by revealing and reasoning about context: socially intelligent agents for enhancing privacy. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), Stockholm, pp. 28–34. External Links: Document Cited by: §6.4.
  • H. Aldewereld, J. Padget, W. Vasconcelos, J. Vazquez-Salceda, P. Sergeant, and A. Staikopoulos (2010) Adaptable, organization-aware, service-oriented computing. IEEE Intelligent Systems 25 (4), pp. 26–35. External Links: Document Cited by: §6.2.
  • C. J. Amaral, J. F. Hübner, and T. Kampik (2020) Towards Jacamo-rest: a resource-oriented abstraction for managing multi-agent systems. In Proceedings of the 14th Workshop-School on Agents, Environments, and Applications, G. Vaz Alves, G. Guiménez Lug, A. Pinz Borgeso, and C. E. Pantoja (Eds.), Ponta Grossa, Parana, Brazil, pp. 140–151. Cited by: §6.2.
  • C. J. Amaral and J. F. Hübner (2020) Jacamo-web is on the fly: an interactive multi-agent system ide. In Engineering Multi-Agent Systems, L. A. Dennis, R. H. Bordini, and Y. Lespérance (Eds.), Cham, pp. 246–255. External Links: ISBN 978-3-030-51417-4 Cited by: §6.2.
  • G. Andrighetto, G. Governatori, P. Noriega, and L. van der Torre (2012) Normative multi-agent systems (Dagstuhl Seminar 12111). Technical report Technical Report 3, Vol. 2, Schloss Dagstuhl-Leibniz-Zentrum fuer InformatikSchloss Dagstuhl-Leibniz-Zentrum fuer Informatik. Cited by: §3.4.
  • A. Ankolekar, M. Burstein, J. R. Hobbs, O. Lassila, D. Martin, D. McDermott, S. A. McIlraith, S. Narayanan, M. Paolucci, T. Payne, and K. Sycara (2002) DAML-s: web service description for the semantic web. In The Semantic Web — ISWC 2002, I. Horrocks and J. Hendler (Eds.), Berlin, Heidelberg, pp. 348–363. External Links: ISBN 978-3-540-48005-1 Cited by: §3.2.
  • J. L. Arcos, M. Esteva, P. Noriega, J. A. Rodríguez-Aguilar, and C. Sierra (2005) Engineering open environments with electronic institutions. Engineering Applications of Artificial Intelligence 18 (2), pp. 191–204. External Links: Document Cited by: item i.
  • J. L. Austin (1962) How to do things with words. William James Lectures, Oxford University Press, Oxford, England. Cited by: §3.2.
  • A. Barker, C. D. Walton, and D. Robertson (2009) Choreographing web services. IEEE Transactions on Services Computing 2 (2), pp. 152–166. External Links: Document Cited by: §4.3.
  • P. Baroni, D. M. Gabbay, M. Giacomin, and L. van der Torre (2018) Handbook of formal argumentation. College Publications, Rickmansworth. Cited by: §5.2.4, §6.3, §6.5.
  • W. Beek, L. Rietveld, H. R. Bazoobandi, J. Wielemaker, and S. Schlobach (2014) LOD laundromat: a uniform way of publishing other people’s dirty data. In The Semantic Web – ISWC 2014, P. Mika, T. Tudorache, A. Bernstein, C. Welty, C. Knoblock, D. Vrandečić, P. Groth, N. Noy, K. Janowicz, and C. Goble (Eds.), Cham, pp. 213–228. External Links: ISBN 978-3-319-11964-9 Cited by: §3.2.
  • F. L. Bellifemine, G. Caire, and D. Greenwood (2007) Developing multi-agent systems with jade (wiley series in agent technology). John Wiley and Sons, Inc., Hoboken, NJ, USA. External Links: ISBN 0470057475 Cited by: §3.1.
  • T. Bench-Capon, H. Prakken, and G. Sartor (2009) Argumentation in legal reasoning. In Argumentation in Artificial Intelligence, pp. 363–382. External Links: ISBN 978-0-387-98197-0, Document, Link Cited by: §5.2.4.
  • T. Berners-Lee, J. Hendler, and O. Lassila (2001) The semantic web. Scientific American 284 (5), pp. 34–43. Cited by: §1, §3.2.
  • G. Boella, J. Hulstijn, and L. van der Torre (2004) Persuasion strategies in dialogue. In Procs. of CMNA Workshop at ECAI’04, Valencia, pp. 29–32. Cited by: §4.3, §6.4.
  • G. Boella, L. Van Der Torre, and H. Verhagen (2006) Introduction to normative multiagent systems. Computational & Mathematical Organization Theory 12 (2-3), pp. 71–79. Cited by: §3.4.
  • O. Boissier, R. H. Bordini, J. Hübner, and A. Ricci (2020) Multi-agent oriented programming: programming multi-agent systems using jacamo. MIT Press, Cambridge. Cited by: §3.1, §3.4, §4.1, §4.1, §4.1, §6.2.
  • P. A. Bonatti, S. Kirrane, I. M. Petrova, and L. Sauro (2020) Machine understandable policies and gdpr compliance checking. KI-Künstliche Intelligenz 34 (3), pp. 303–315. Cited by: §3.4.
  • P. A. Bonatti and D. Olmedilla (2007) Rule-based policy representation and reasoning for the semantic web. In Reasoning Web: Third International Summer School 2007, Dresden, Germany, September 3-7, 2007, Tutorial Lectures, pp. 240–268. External Links: ISBN 978-3-540-74615-7, Document Cited by: §3.4.
  • R. H. Bordini, J. F. Hübner, and M. Wooldridge (2007) Programming multi-agent systems in agentspeak using jason (wiley series in agent technology). John Wiley & Sons, Inc., Hoboken, NJ, USA. External Links: ISBN 0470029005 Cited by: §3.1, §3.1, §4.2.2, §6.6.
  • J. M. Bradshaw (Ed.) (1997) Software agents. MIT press, Cambridge, MA. Cited by: §3.4.
  • L. Braubach, A. Pokahr, and W. Lamersdorf (2005) Jadex: a bdi-agent system combining middleware and reasoning. In Software Agent-Based Applications, Platforms and Development Kits, R. Unland, M. Calisti, and M. Klusch (Eds.), Basel, pp. 143–168. External Links: ISBN 978-3-7643-7348-1 Cited by: §3.1.
  • T. Bray, J. Paoli, C. M. Sperberg-McQueen, E. Maler, and F. Yergeau (2008) Extensible Markup Language (XML) 1.0 (Fifth Edition), W3C Recommendation 26 November 2008. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3.
  • D. Brickley and R. V. Guha (2014) RDF Schema 1.1, W3C Recommendation 25 February 2014. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.2.
  • R. A. Brooks (1991) Intelligence without representation. Artificial Intelligence 47 (1), pp. 139–159. External Links: ISSN 0004-3702 Cited by: §4.2.2.
  • J. Bryson and L. A. Stein (2001) Modularity and design in reactive intelligence. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, IJCAI 2001, Seattle, Washington, USA, August 4-10, 2001, B. Nebel (Ed.), Palo Alto, California, pp. 1115–1120. Cited by: §4.2.2.
  • P. Busetta, R. Rönnquist, A. Hodgson, and A. Lucas (2019) JACK intelligent agents - components for intelligent agents in java. Note: Available from Newsletter Issue 2 Cited by: §3.1.
  • E. Cabrio, A. Palmero Aprosio, and S. Villata (2014)

    These Are Your Rights - A Natural Language Processing Approach to Automated RDF Licenses Generation

    In The Semantic Web: Trends and Challenges - 11th International Conference, ESWC 2014, Anissaras, Crete, Greece, May 25-29, 2014. Proceedings, Lecture Notes in Computer Science, Vol. 8465, Cham, pp. 255–269. External Links: ISBN 978-3-319-07442-9 Cited by: §3.4.
  • R. Calegari, G. Ciatto, V. Mascardi, and A. Omicini (2021) Logic-based technologies for multi-agent systems: a systematic literature review. Autonomous Agents and Multi Agent Systems 35 (1), pp. 1. External Links: Document Cited by: §4.2.2.
  • F. Caron, J. Vanthienen, and B. Baesens (2013) Comprehensive rule-based compliance checking and risk management with process mining. Decision Support Systems 54 (3), pp. 1357–1369. External Links: ISSN 0167-9236, Document, Link Cited by: §4.3.
  • A. K. Chopra, S. H. Christie V, and M. P. Singh (2020) An evaluation of communication protocol languages for engineering multiagent systems. Journal of Artificial Intelligence Research (JAIR) 69, pp. 1351–1393. External Links: Document Cited by: §6.1.
  • A. K. Chopra and M. P. Singh (2016) From social machines to social protocols: software engineering foundations for sociotechnical systems. In Proceedings of the 25th International World Wide Web Conference, Montréal, pp. 903–914. External Links: Document Cited by: §6.2.
  • A. Chopra and M. P. Singh (2004) Nonmonotonic commitment machines. In Advances in Agent Communication: Proceedings of the International Workshop on Agent Communication Languages (ACL 2003), F. Dignum (Ed.), Lecture Notes in Artificial Intelligence, Melbourne, pp. 183–200. External Links: Document Cited by: §6.1.
  • A. Chopra, L. van der Torre, H. Verhagen, and S. Villata (2018) Handbook of normative multiagent systems. College Publications, Rickmansworth. External Links: ISBN 9781848902855 Cited by: §3.4.
  • A. Cimmino, M. McCool, F. Tavakolizadeh, and K. Toumura (2020) Web of Things (WoT) Discovery, W3C First Public Working Draft 24 November 2020. W3C First Public Working Draft World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3.
  • A. Ciortea, O. Boissier, and A. Ricci (2018a) Engineering world-wide multi-agent systems with hypermedia. In Engineering Multi-Agent Systems - 6th International Workshop, EMAS 2018, Stockholm, Sweden, July 14-15, 2018, Revised Selected Papers, D. Weyns, V. Mascardi, and A. Ricci (Eds.), Lecture Notes in Computer Science, Vol. 11375, Cham, pp. 285–301. External Links: Document Cited by: §3.3, §6.6.
  • A. Ciortea, O. Boissier, A. Zimmermann, and A. M. Florea (2018b) Give agents some rest: hypermedia-driven agent environments. In Engineering Multi-Agent Systems, A. El Fallah-Seghrouchni, A. Ricci, and T. C. Son (Eds.), Cham, pp. 125–141. External Links: ISBN 978-3-319-91899-0 Cited by: §3.3, §6.6.
  • A. Ciortea, S. Mayer, O. Boissier, and F. Gandon (2019) Exploiting Interaction Affordances: On Engineering Autonomous Systems for the Web of Things. In Second W3C Workshop on the Web of Things The Open Web to Challenge IoT Fragmentation, Munich, Germany, pp. 0–4. External Links: Link Cited by: §3.3, §6.6.
  • A. Ciortea, S. Mayer, F. Gandon, O. Boissier, A. Ricci, and A. Zimmermann (2019) A Decade in Hindsight: The Missing Bridge Between Multi-Agent Systems and the World Wide Web. In AAMAS 2019 - 18th International Conference on Autonomous Agents and Multiagent Systems, Montréal, Canada, pp. 5. External Links: Link Cited by: §3.2, §3.3, §3.
  • S. Cox and C. Little (2017) The Time Ontology in OWL, W3C Recommendation 19 October 2017. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3.
  • S. Cranefield, M. Winikoff, V. Dignum, and F. Dignum (2017) No pizza for you: value-based plan selection in BDI agents. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, (IJCAI), C. Sierra (Ed.), Melbourne, pp. 178–184. External Links: Document Cited by: item iii.
  • L. Cranor, M. Langheinrich, M. Marchiori, M. Presler-Marshall, and J. Reagle (2002) The platform for privacy preferences 1.0 (p3p1.0) specification. W3C Recommendation World Wide Web Consortium. Note: Obsolete recommendation since 30 August 2018 Cited by: §3.4.
  • N. Criado, E. Argente, P. Noriega, and V. Botti (2010) Towards a normative bdi architecture for norm compliance. In COIN@, Lyon, France, pp. 65. Cited by: §3.4.
  • R. Cyganiak, D. Wood, and M. Lanthaler (2014) RDF 1.1 Concepts and Abstract Syntax, W3C Recommendation 25 February 2014. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3.
  • M. d’Inverno, M. Luck, P. Noriega, J. A. Rodríguez-Aguilar, and C. Sierra (2012) Communicating open systems. Artificial Intelligence 186, pp. 38–94. External Links: Document Cited by: §3.4.
  • E. Daga, M. d’Aquin, A. Adamou, and E. Motta (2016) Addressing exploitability of smart city data. In 2016 IEEE International Smart Cities Conference (ISC2), Vol. , Piscataway, New Jersey, United States, pp. 1–6. External Links: Document Cited by: §3.4.
  • E. Daga, M. d’Aquin, A. Gangemi, and E. Motta (2015) Propagation of policies in rich data flows. In Proceedings of the 8th International Conference on Knowledge Capture, K-CAP 2015, New York, NY, USA. External Links: ISBN 9781450338493, Link, Document Cited by: §3.4.
  • F. Dalpiaz, J. Dix, and M. B. van Riemsdijk (Eds.) (2014) Engineering multi-agent systems - second international workshop, EMAS 2014, paris, france, may 5-6, 2014, revised selected papers. Lecture Notes in Computer Science, Vol. 8758, Springer, Cham. External Links: Document, ISBN 978-3-319-14483-2 Cited by: J. Lee, J. Padget, B. Logan, D. Dybalova, and N. Alechina (2014).
  • M. Dastani, J. Dix, H. Verhagen, and S. Villata (2018) Normative multi-agent systems (dagstuhl seminar 18171). Dagstuhl Reports 8 (4), pp. 72–103. Cited by: §3.4.
  • M. Dastani (2008) 2APL: a practical agent programming language. Autonomous Agents and Multi-Agent Systems 16 (3), pp. 214–248. Cited by: §3.1.
  • F. De la Prieta, S. Rodríguez-González, P. Chamoso, J. M. Corchado, and J. Bajo (2019) Survey of agent-based cloud computing applications. Future Generation Computer Systems 100, pp. 223–236. External Links: Document, ISSN 0167739X, Link Cited by: §3.1.
  • M. De Vos, S. Kirrane, J. Padget, and K. Satoh (2019) ODRL policy modelling and compliance checking. In Rules and Reasoning, P. Fodor, M. Montali, D. Calvanese, and D. Roman (Eds.), Cham, pp. 36–51. External Links: ISBN 978-3-030-31095-0 Cited by: §3.4.
  • K. Decker, K. Sycara, and M. Williamson (1997) Middle-agents for the internet. In IJCAI-97: Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, San Francisco, California, pp. 578–583. Cited by: §3.2.
  • N. Desai, A. U. Mallya, A. K. Chopra, and M. P. Singh (2006) OWL-P: a methodology for business process development. In Agent-Oriented Information Systems III, 7th International Bi-Conference Workshop, AOIS2005, Utrecht, Netherlands, July 26, 2005, and Klagenfurt, Austria, October 27, 2005, Revised Selected Papers, Lecture Notes in Computer Science, Berlin, pp. 79–94. External Links: Document Cited by: §3.2.
  • M. J. Dürst and M. Suignard (2005) Internationalized Resource Identifiers (IRIs). Technical report Internet Engineering Task Force. External Links: Link Cited by: footnote 16.
  • M. Eisenstadt, M. Ramachandran, N. Chowdhury, A. Third, and J. Domingue (2020) COVID-19 antibody test/vaccination certification: there’s an app for that. IEEE Open Journal of Engineering in Medicine and Biology 1 (), pp. 148–155. External Links: Document Cited by: §2.
  • M. Esteva, J. Padget, and C. Sierra (2002) Formalizing a language for institutions and norms. In Intelligent Agents VIII, J. Ch. Meyer and M. Tambe (Eds.), Berlin, Heidelberg, pp. 348–366. External Links: ISBN 978-3-540-45448-9 Cited by: §3.4.
  • D. Fensel and E. Motta (2001) Structured development of problem solving methods. IEEE Trans. Knowl. Data Eng. 13 (6), pp. 913–932. External Links: Document Cited by: §3.2.
  • J. Ferber, O. Gutknecht, and F. Michel (2004) From agents to organizations: an organizational view of multi-agent systems. In Agent-Oriented Software Engineering IV, P. Giorgini, J. P. Müller, and J. Odell (Eds.), Berlin, Heidelberg, pp. 214–230. Cited by: §3.1, §4.2.3.
  • A. Ferrando, M. Winikoff, S. Cranefield, F. Dignum, and V. Mascardi (2019) On enactability of agent interaction protocols: towards a unified approach. In Proceedings of the 7th International Workshop on Engineering Multi-Agent Systems (EMAS), Lecture Notes in Computer Science, Vol. 12058, Montréal, pp. 43–64. External Links: Document Cited by: §4.3, §6.1.
  • R. T. Fielding and J. Reschke (2014) Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content. Technical report IETF. External Links: Link Cited by: §3.3.
  • R. T. Fielding (2000) Representational State Transfer (REST). Ph.D. Thesis, University of California, Irvine. Note: Chapter 5 of Roy Fielding’s thesis Cited by: §3.3.
  • T. Finin, R. Fritzson, D. McKay, and R. McEntire (1994) KQML as an agent communication language. In Proceedings of the Third International Conference on Information and Knowledge Management, CIKM ’94, New York, NY, USA, pp. 456–463. External Links: ISBN 0897916743, Document Cited by: §3.2.
  • FIPA (2003) FIPA interaction protocol specifications. Note: FIPA: The Foundation for Intelligent Physical Agents External Links: Link Cited by: §3.1, §6.1.
  • N. Fornara and M. Colombetti (2019) Using semantic web technologies and production rules for reasoning on obligations, permissions, and prohibitions. AI Communications 32, pp. 319–334. External Links: Document, ISBN 1875-8452 Cited by: §3.4, §4.1.
  • D. Z. G. Garcia and M. B. F. de Toledo (2008) A web service privacy framework based on a policy approach enhanced with ontologies. In 2008 11th IEEE International Conference on Computational Science and Engineering-Workshops, Piscataway, New Jersey, United States, pp. 209–214. Cited by: §3.4.
  • G. Governatori, L. Ho-Pun, A. Rotolo, S. Villata, and F. Gandon (2013a) Heuristics for Licenses Composition. In Frontiers in Artificial Intelligence and Applications, Vol. 259, Amsterdam, The Netherlands, pp. 77–86. Cited by: §3.4.
  • G. Governatori, H. Lam, A. Rotolo, S. Villata, G. A. Atemezing, and F. Gandon (2014) LIVE: a tool for checking licenses compatibility between vocabularies and data. In Proceedings of the 2014 International Conference on Posters & Demonstrations Track - Volume 1272, ISWC-PD’14, Aachen, DEU, pp. 77–80. Cited by: §3.4.
  • G. Governatori, A. Rotolo, S. Villata, and F. Gandon (2013b) One license to compose them all. In The Semantic Web – ISWC 2013, H. Alani, L. Kagal, A. Fokoue, P. Groth, C. Biemann, J. X. Parreira, L. Aroyo, N. Noy, C. Welty, and K. Janowicz (Eds.), Berlin, Heidelberg, pp. 151–166. External Links: ISBN 978-3-642-41335-3 Cited by: §3.4.
  • W. O. W. Group (2012) OWL 2 Web Ontology Language Document Overview (Second Edition), W3C Recommendation 11 December 2012. Technical report World Wide Web Consortium (W3C) (en). External Links: Link Cited by: §3.2, §3.3.
  • M. Gueroussova, A. Polleres, and S. McIlraith (2013) SPARQL with qualitative and quantitative preferences. In Proceedings of the 2nd International Conference on Ordering and Reasoning - Volume 1059, OrdRing’13, Aachen, DEU, pp. 2–8. Cited by: §3.2.
  • A. Haller, K. Janowicz, S. Cox, D. Le Phuoc, J. Taylor, and M. Lefrançois (2017) Semantic Sensor Network Ontology, W3C Recommendation 19 October 2017. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3.
  • S. Harris and A. Seaborne (2013) SPARQL 1.1 Query Language, W3C Recommendation 21 March 2013. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.2.
  • C. Haynes, M. Luck, P. McBurney, S. Mahmoud, T. Vitek, and S. Miles (2017) Engineering the emergence of norms: a review. Knowledge Eng. Review 32, pp. e18. Cited by: §3.4.
  • High-Level Expert Group on Artificial Intelligence (AI HLEG) (2019) Ethics Guidelines for Trustworthy AI. External Links: Link Cited by: §1, §3.4, §6.4.
  • K. V. Hindriks and J. Dix (2014) GOAL: a multi-agent programming language applied to an exploration game. In Agent-Oriented Software Engineering, O. Shehory and A. Sturm (Eds.), Cham, pp. 235–258 (English). External Links: Document, ISBN 978-3-642-54431-6 Cited by: §3.1.
  • M. N. Huhns, N. Jacobs, T. Ksiezyk, W. Shen, M. P. Singh, and P. E. Cannata (1993) Integrating enterprise information models in Carnot. In Proceedings of the International Conference on Intelligent and Cooperative Information Systems (ICICIS), Rotterdam, pp. 32–42. External Links: Document Cited by: §3.2.
  • M. N. Huhns and M. P. Singh (1997) Ontologies for agents. IEEE Internet Computing (IC) 1 (6), pp. 81–83. Note: Instance of the column Agents on the Web External Links: Document Cited by: §3.2.
  • R. Ianella and S. Villata (2018) ODRL Information Model 2.2, W3C Recommendation 15 February 2018. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.4.
  • I. Jacobs and N. Walsh (2004) Architecture of the World Wide Web, Volume One, W3C Recommendation 15 December 2004. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3, §4.1, §4.1.
  • S. Kaebisch, T. Kamiya, M. McCool, V. Charpenay, and M. Kovatsch (2020) Web of Things (WoT) Thing Description, W3C Recommendation 9 April 2020. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3, §3.3.
  • Ö. Kafali, N. Ajmeri, and M. P. Singh (2020) DESEN: specification of sociotechnical systems via patterns of regulation and control. ACM Transactions on Software Engineering and Methodology (TOSEM) 29 (1). External Links: ISSN 1049-331X, Document Cited by: §2, §5.2.4.
  • L. Kagal, T. Finin, and A. Joshi (2003) A policy language for a pervasive computing environment. In Proceedings of the 4th IEEE International Workshop on Policies for Distributed Systems and Networks, POLICY ’03, USA, pp. 63. External Links: ISBN 0769519334 Cited by: §3.4.
  • T. Kampik, A. Gomez, A. Ciortea, and S. Mayer (2021) Autonomous agents on the edge of things. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1767–1769. External Links: ISBN 9781450383073 Cited by: §4.1.
  • M. Kifer, G. Lausen, and J. Wu (1995) Logical foundations of object-oriented and frame-based languages. J. ACM 42 (4), pp. 741–843. External Links: ISSN 0004-5411, Document Cited by: §3.2.
  • S. Kirrane and S. Decker (2018) Intelligent agents: the vision revisited. In Proceedings of the 2nd Workshop on Decentralizing the Semantic Web co-located with the 17th International Semantic Web Conference, Aachen, Germany, pp. 9–17. Cited by: §3.4.
  • S. Kirrane, A. Mileo, and S. Decker (2017) Access control and the resource description framework: a survey. Semantic Web 8 (2), pp. 311–352. Cited by: §3.4.
  • S. Kirrane (2021) Intelligent software web agents: a gap analysis. External Links: 2102.06607 Cited by: §3.2.
  • Z. Kis, D. Peinter, C. Aguzzi, J. Hund, and K. Nimura (2020) Web of Things (WoT) Scripting API, W3C Working Group Note 24 November 2020. W3C Working Group Note World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3, §3.3, §6.6.
  • P. Kolari, L. Ding, G. Shashidhara, A. Joshi, T. Finin, and L. Kagal (2005) Enhancing web privacy protection through declarative policies. In Sixth IEEE International Workshop on Policies for Distributed Systems and Networks (POLICY’05), Piscataway, New Jersey, United States, pp. 57–66. Cited by: §3.4.
  • M. Kovatsch, R. Matsukura, M. Lagally, T. Kawaguchi, K. Toumura, and K. Kajimoto (2020) Web of Things (WoT) Architecture, W3C Recommendation 9 April 2020. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3, §3.3, §6.6.
  • K. Kravari and N. Bassiliades (2015) A survey of agent platforms. Journal of Artificial Societies and Social Simulation 18 (1), pp. 11. Cited by: §3.1.
  • T. Lebo, S. S. Sahoo, and D. L. McGuinness (2013) PROV-O: The PROV Ontology, W3C Recommendation 30 April 2013. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.3.
  • J. Lee, J. Padget, B. Logan, D. Dybalova, and N. Alechina (2014) N-Jason: run-time norm compliance in AgentSpeak(L). See Engineering multi-agent systems - second international workshop, EMAS 2014, paris, france, may 5-6, 2014, revised selected papers, Dalpiaz et al., pp. 367–387. External Links: Document Cited by: §3.4, §3.4.
  • D. Lewis (1969) Convention — a philosophical study. Harvard University Press, Cambridge, MA. Cited by: §3.4.
  • B. Light, C. P. Holland, and K. Wills (2001) ERP and best of breed: a comparative analysis. Business Process Management Journal 7 (3), pp. 216–224. External Links: Document, ISBN 1463-7154, Link Cited by: §6.6.
  • B. Logan (2018) An agent programming manifesto. International Journal of Agent-Oriented Software Engineering 6 (2), pp. 187–210. Cited by: §6.6.
  • A. U. Mallya and M. P. Singh (2007) An algebra for commitment protocols. Journal of Autonomous Agents and Multi-Agent Systems (JAAMAS) 14 (2), pp. 143–163. External Links: Document Cited by: §6.1.
  • D. Martin, M. Paolucci, S. McIlraith, M. Burstein, D. McDermott, D. McGuinness, B. Parsia, T. Payne, M. Sabou, M. Solanki, N. Srinivasan, and K. Sycara (2005) Bringing semantics to web services: the owl-s approach. In Semantic Web Services and Web Process Composition, J. Cardoso and A. Sheth (Eds.), Berlin, Heidelberg, pp. 26–42. External Links: ISBN 978-3-540-30581-1 Cited by: §3.2.
  • V. Mascardi, D. Weyns, A. Ricci, C. B. Earle, A. Casals, M. Challenger, A. Chopra, A. Ciortea, L. A. Dennis, Á. F. Díaz, A. El Fallah-Seghrouchni, A. Ferrando, L. Fredlund, E. Giunchiglia, Z. Guessoum, A. Günay, K. Hindriks, C. A. Iglesias, B. Logan, T. Kampik, G. Kardas, V. J. Koeman, J. B. Larsen, S. Mayer, T. Méndez, J. C. Nieves, V. Seidita, B. T. Teze, L. Z. Varga, and M. Winikoff (2019) Engineering multi-agent systems: state of affairs and the road ahead. SIGSOFT Softw. Eng. Notes 44 (1), pp. 18–28. External Links: ISSN 0163-5948, Document Cited by: §3.1, §6.6.
  • J. Morales, M. López-Sánchez, J. A. Rodríguez-Aguilar, W. W. Vasconcelos, and M. J. Wooldridge (2015) Online automated synthesis of compact normative systems. ACM Trans. Auton. Adapt. Syst. 10 (1), pp. 2:1–2:33. External Links: Link, Document Cited by: item i.
  • A. Morris-Martin, M. De Vos, and J. Padget (2019) Norm emergence in multiagent systems: a viewpoint paper. Autonomous Agents and Multi-Agent Systems 33 (6), pp. 706–749. External Links: Document, ISBN 1573-7454 Cited by: §3.4, §6.4.
  • A. Morris-Martin, M. De Vos, and J. Padget (2021) A norm emergence framework for normative mas – position paper. In Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XIII, A. Aler Tubella, S. Cranefield, C. Frantz, F. Meneguzzi, and W. Vasconcelos (Eds.), Cham, pp. 156–174. External Links: ISBN 978-3-030-72376-7 Cited by: §3.4.
  • P. Noriega (1997) Agent mediated auctions: the fishmarket metaphor. Ph.D. Thesis, Universitat Autònoma de Barcelona, (English). External Links: Link Cited by: §3.4, §3.4.
  • N. Osman, R. Chenu-Abente, Q. Shen, C. Sierra, and F. Giunchiglia (2021) Empowering users in online open communities. SN Comput. Sci. 2 (4), pp. 338. External Links: Link, Document Cited by: item iii.
  • S. Ossowski (2012) Agreement technologies. Springer Publishing Company, Incorporated, Berlin. External Links: ISBN 9400755821 Cited by: §6.5.
  • E. Ostrom (1990) Governing the commons. the evolutions of institutions for collective action.. Cambridge University Press, Cambridge. Cited by: §3.4.
  • J. Padget, M. De Vos, and C. A. Page (2018) Deontic sensors. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI’18, Palo Alto, pp. 475–481. External Links: ISBN 9780999241127 Cited by: §3.4.
  • M. Palmirani, G. Governatori, A. Rotolo, S. Tabet, H. Boley, and A. Paschke (2011) LegalRuleML: xml-based rules and norms. In Proceedings of the 5th International Conference on Rule-Based Modeling and Computing on the Semantic Web, RuleML’11, Berlin, Heidelberg, pp. 298–312. External Links: ISBN 9783642249075 Cited by: §3.4.
  • M. Paolucci, T. Kawamura, T. R. Payne, and K. Sycara (2002) Semantic matching of web services capabilities. In The Semantic Web — ISWC 2002, I. Horrocks and J. Hendler (Eds.), Berlin, Heidelberg, pp. 333–347. External Links: ISBN 978-3-540-48005-1 Cited by: §3.2, §6.2.
  • S. Parsons (2001) Qualitative methods for reasoning under uncertainty. MIT Press, Cambridge, Massachusetts, United States. External Links: ISBN 978-0-262-16168-8 Cited by: §6.3.
  • P. F. Patel-Schneider, A. Polleres, and D. Martin (2018) Comparative preferences in SPARQL. In Knowledge Engineering and Knowledge Management - 21st International Conference, EKAW 2018, Nancy, France, November 12-16, 2018, Proceedings, C. Faron-Zucker, C. Ghidini, A. Napoli, and Y. Toussaint (Eds.), Lecture Notes in Computer Science, Vol. 11313, Cham, pp. 289–305. External Links: Link, Document Cited by: §3.2.
  • O. Pivert, O. Slama, and V. Thion (2016) SPARQL extensions with preferences: a survey. In Proceedings of the 31st Annual ACM Symposium on Applied Computing, SAC ’16, New York, NY, USA, pp. 1015–1020. External Links: ISBN 9781450337397, Link, Document Cited by: §4.1.
  • M. Poveda-Villalón, A. Gómez-Pérez, and M. C. Suárez-Figueroa (2014) OOPS! (OntOlogy Pitfall Scanner!): An On-line Tool for Ontology Evaluation. International Journal on Semantic Web and Information Systems (IJSWIS) 10 (2), pp. 7–34. Cited by: §3.2.
  • I. Rahwan and C. Reed (2009) The argument interchange format. In Argumentation in Artificial Intelligence, pp. 383–402. External Links: ISBN 978-0-387-98197-0, Document, Link Cited by: §6.5.
  • A. S. Rao and M. P. Georgeff (1995) BDI agents: from theory to practice. In Proceedings of the First International Conference on Multiagent Systems, June 12-14, 1995, San Francisco, California, USA, V. R. Lesser and L. Gasser (Eds.), Cambridge, pp. 312–319. Cited by: §4.2.2.
  • W. Reisig (1985) Petri nets: an introduction. Springer-Verlag, Berlin, Heidelberg. External Links: ISBN 0387137238 Cited by: §4.3.
  • E. Reshetova and M. McCool (2019) Web of Things (WoT) Security and Privacy Guidelines, W3C Working Group Note 6 November 2019. W3C Working Group Note World Wide Web Consortium (W3C). External Links: Link Cited by: §3.4.
  • A. Rhayem, M. B. A. Mhiri, and F. Gargouri (2020) Semantic web technologies for the internet of things: systematic literature review. Internet Things 11, pp. 100206. External Links: Link, Document Cited by: §3.3.
  • D. Roman, U. Keller, H. Lausen, J. de Bruijn, R. Lara, M. Stollberg, A. Polleres, C. Feier, C. Bussler, and D. Fensel (2005) Web service modeling ontology. Appl. Ontology 1 (1), pp. 77–106. External Links: Link Cited by: §3.2.
  • A. Sambra and S. Corlosquet (2015) WebID 1.0 - Web Identity and Discovery, W3C Editor’s Draft 28 May 2014. W3C Editor’s draft World Wide Web Consortium (W3C). External Links: Link Cited by: §3.4.
  • M. Sensoy, T. J. Norman, W. W. Vasconcelos, and K. Sycara (2012) OWL-polar: a framework for semantic policy representation and reasoning. Journal of Web Semantics 12-13, pp. 148–160. Note: Reasoning with context in the Semantic Web External Links: ISSN 1570-8268, Document, Link Cited by: §3.4, §4.1.
  • Z. Shams, M. D. Vos, N. Oren, and J. A. Padget (2020) Argumentation-based reasoning about plans, maintenance goals, and norms. ACM Trans. Auton. Adapt. Syst. 14 (3), pp. 9:1–9:39. External Links: Link, Document Cited by: item ii, §3.4, §3.4.
  • M. P. Singh, A. K. Chopra, and N. Desai (2009) Commitment-based service-oriented architecture. Computer 42 (11), pp. 72–79. External Links: Document Cited by: §2, §5.2.4.
  • M. P. Singh and A. K. Chopra (2017) The internet of things and multiagent systems: decentralized intelligence in distributed computing. In 37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017, Atlanta, GA, USA, June 5-8, 2017, K. Lee and L. Liu (Eds.), Washington, D.C., United States, pp. 1738–1747. External Links: Link, Document Cited by: §3.3.
  • M. P. Singh and A. K. Chopra (2020) Clouseau: generating communication protocols from commitments. In Proceedings of the 34th Conference on Artificial Intelligence (AAAI), New York, pp. 7244–7252. External Links: Document Cited by: §6.1.
  • M. P. Singh and S. H. Christie V (2021) Tango: declarative semantics for multiagent communication protocols. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI), Online, pp. 391–397. External Links: Document Cited by: §6.1.
  • M. P. Singh and M. N. Huhns (2005) Service-oriented computing: semantics, processes, agents. John Wiley & Sons, Chichester, United Kingdom. External Links: Document Cited by: §1.
  • M. P. Singh (2011) Information-driven interaction-oriented programming: BSPL, the Blindingly Simple Protocol Language. In Proceedings of the 10th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), Taipei, pp. 491–498. External Links: Document Cited by: §3.1, §4.3, §6.1.
  • M. P. Singh (2013) Norms as a basis for governing sociotechnical systems. ACM Transactions on Intelligent Systems and Technology (TIST) 5 (1). External Links: ISSN 2157-6904, Document Cited by: §3.4.
  • S. Speicher, J. Arwe, and A. Malhotra (2015) Linked Data Platform 1.0, W3C Recommendation 26 February 2015. W3C Recommendation World Wide Web Consortium (W3C). External Links: Link Cited by: §3.2.
  • S. Staab and H. Stuckenschmidt (Eds.) (2006) Semantic web and peer-to-peer - decentralized management and exchange of knowledge and information. Springer, Berlin. External Links: Document, ISBN 978-3-540-28346-1 Cited by: §3.2.
  • S. Steyskal and S. Kirrane (2015) If you can’t enforce it, contract it: enforceability in policy-driven (linked) data markets. In Semantics (posters & demos), Aachen, pp. 63–66. Cited by: §3.4.
  • S. Steyskal and A. Polleres (2014) Defining expressive access policies for linked data using the odrl ontology 2.0. In Proceedings of the 10th International Conference on Semantic Systems, SEM ’14, New York, NY, USA, pp. 20–23. External Links: ISBN 9781450329279, Link, Document Cited by: §3.4.
  • C. Straßer and O. Arieli (2014) Sequent-based argumentation for normative reasoning. In Deontic Logic and Normative Systems, F. Cariani, D. Grossi, J. Meheus, and X. Parent (Eds.), Cham, pp. 224–240. External Links: ISBN 978-3-319-08615-6 Cited by: §3.4.
  • S. Tanwar, K. Parekh, and R. Evans (2020) Blockchain-based electronic healthcare record system for healthcare 4.0 applications. Journal of Information Security and Applications 50, pp. 102407. External Links: ISSN 2214-2126, Document Cited by: §2, §2.
  • P. R. Telang, M. P. Singh, and N. Yorke-Smith (2019) A coupled operational semantics for goals and commitments. Journal of Artificial Intelligence Research (JAIR) 65, pp. 31–85. External Links: Document Cited by: §3.4.
  • A. Theodorou, R. H. Wortham, and J. J. Bryson (2017) Designing and implementing transparency for real time inspection of autonomous robots. Connect. Sci. 29 (3), pp. 230–241. External Links: Link, Document Cited by: §4.2.2.
  • A. Tour, A. Polyvyanyy, and A. Kalenkova (2021) Agent system mining: vision, benefits, and challenges. IEEE Access 9, pp. 99480–99494. Cited by: §4.3.
  • Y. B. Udupi and M. P. Singh (2006) Contract enactment in virtual organizations: a commitment-based approach. In Proceedings of the 21st National Conference on Artificial Intelligence (AAAI), Boston, pp. 722–727. Cited by: §3.4.
  • W. Van Der Aalst (2012) Process mining. Communications of the ACM 55 (8), pp. 76–83. Cited by: §4.3.
  • M. B. van Riemsdijk, L. A. Dennis, M. Fisher, and K. V. Hindriks (2013) Agent reasoning for norm compliance: a semantic approach. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS ’13, Richland, SC, pp. 499–506. External Links: ISBN 9781450319935 Cited by: §3.4.
  • R. Verborgh (2018) Decentralizing the Semantic Web through incentivized collaboration. In Proceedings of the 17th International Semantic Web Conference: Blue Sky Track, M. van Erp, M. Atre, V. Lopez, K. Srinivas, and C. Fortuna (Eds.), CEUR Workshop Proceedings, Vol. 2189, Aachen, DEU, pp. 1–5. External Links: ISSN 1613-0073 Cited by: §2.
  • S. Villata and F. Gandon (2012) Licenses compatibility and composition in the web of data. In Proceedings of the Third International Conference on Consuming Linked Data - Volume 905, COLD’12, Aachen, DEU, pp. 124–135. Cited by: §3.4.
  • D. Walton and E.C.W. Krabbe (1995) Commitment in dialogue: basic concepts of interpersonal reasoning. SUNY series in Logic and Language, State University of New York Press, New York, NY, USA. External Links: ISBN 9780791425862, LCCN lc94037799 Cited by: §4.3.
  • G. Weiss (1999) Multiagent systems: a modern approach to distributed artificial intelligence. MIT press, Cambridge, Massachusetts, United States. Cited by: §3.1.
  • D. Weyns, A. Omicini, and J. Odell (2007) Environment as a first class abstraction in multiagent systems. Journal of Autonomous Agents and Multi-Agent Systems (JAAMAS) 14 (1), pp. 5–30. Cited by: §3.1.
  • M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J. Boiten, L. B. da Silva Santos, P. E. Bourne, et al. (2016) The fair guiding principles for scientific data management and stewardship. Scientific data 3 (1), pp. 1–9. Cited by: §3.4.
  • M. J. Wooldridge (2001) Introduction to multiagent systems. John Wiley & Sons, Inc., USA. External Links: ISBN 047149691X Cited by: §3.4, §3.4.
  • M. Wooldridge and N. R. Jennings (1995) Intelligent agents: theory and practice. Knowledge engineering review 10 (2), pp. 115–152. Cited by: §4.2.2.