Building secure distributed applications the DECENT way

04/04/2020 ∙ by Haofan Zheng, et al. ∙ University of California Santa Cruz 0

Remote attestation (RA) enables distributed applications that deploy trusted code to enclaves on untrusted hosts and authenticate these components remotely. However, trust relationships established by one component may impact the security of the other components that rely on it, making it difficult to reason about the end-to-end security of these applications. Furthermore, traditional RA approaches interact badly with modern web service design, which tends to employ small interacting microservices, short session lifetimes, and little or no state. This paper presents the Decent Application Platform, a framework for building secure decentralized applications. Decent applications authenticate and authorize distributed components using a protocol based on self-attestation certificates, a reusable credential based on remote attestation and verifiable by a third party. Decent components are authenticated not only based on their code, but also based on the other components they trust, ensuring that no transitively-connected components receive unauthorized information. We evaluate the expressiveness and performance of Decent with two applications: DecentRide, a ride-sharing service, and DecentHT, a distributed hash table. On the YCSB benchmark, we show that DecentHT achieves 7.5x higher throughput and 3.67x lower latency compared to a non-Decent implementation.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

A fundamental challenge in building secure decentralized applications is that untrustworthy nodes may arbitrarily deviate from their expected behavior. A remote service may appear to execute a well-known system component when in fact it is executing a maliciously modified version. Trusted execution environments (TEEs) partially address this challenge by shifting trust from the host executing the component to the manufacturer of the host’s hardware. By provisioning unique private keys to each TEE and certifying the corresponding public keys, third parties can authenticate messages produced within a genuine TEE as long as the key is kept secret.111Doing so is not necessarily trivial: implementation errors, side channels, and physical attacks could potentially leak these keys. We assume the security of TEE platforms for the purposes of this paper, even though that is demonstrably untrue for Intel SGX (e.g., [9]). Of particular interest are messages that attest to what code is executing within the TEE. These messages, called remote attestations, allow a remote node to prove it is executing an authentic system component.

Authenticating a TEE typically requires trust in a centralized entity such as the chip manufacturer, but given the identity established by this process, authorizing a TEEs to access protected resources is determined by whatever entities control those resources. In a decentralized TEE application, mutually distrustful entities may wish to protect their resources within the same application, and may disagree on which entities are trusted. Conceptually, remote attestation places trust in code at the center of a distributed application’s security. Rather than consider whether a host will execute a component faithfully, developers can focus on the intrinsic behavior of the component to ensure their application behaves as expected.

Current TEE frameworks force programmers to work at the wrong level of abstraction where they must deal with many low-level protocol details. These frameworks work fine for simple scenarios where one host wishes to authenticate remote code running on another host, such as when a server wishes to authenticate client code, as illustrated in Figure 0(a), or a client wishes to authenticate server code, as illustrated in Figure 0(b). To authenticate a server component, the server attests to a cryptographic hash of the code it is executing, signs it with its private keys and sends it to the client. The client verifies the signature to ensure the message originated from an authentic TEE and then compares the hash to an expected value.

Even modest extensions of this scenario introduce challenges. Consider the scenario, illustrated in Figure 2, where two components wish to mutually authenticate each other. Each TEE attests to a cryptographic hash of the code it is executing and sends the signed attestation to the other host. Authenticating one component to the other is subtly different than Figure 1 because the verifying component must know what hash value to expect from the remote host, otherwise, users will only know the enclave will communicate with some other enclave but do not know what specific enclave it is. Hardcoding this value in the component is not possible for both components because it introduces a circular dependency: the hash of one component depends on the hash of the other component. Hashing only the enclave code excluding the hash of the other component does not solve the problem, since users only know the enclave is going to communication wit some enclave, but do not know what specific enclave it is connecting to. Moreover, using a value provided by the local host is insecure since (unlike in Figure 1) the local host of the verifying component is untrusted. A malicious host might provide the hash of a malicious component that leaks the message to an adversary.

(a) Enclaves attest to a server.
(b) Enclave attests to clients.
Fig. 1: Traditional RA authentication
Fig. 2: Mutual RA authentication. Enclave A and B cannot hardcode each other’s identity directly in their code since it creates a circular dependency.

A trustworthy hash to authenticate a remote component may not be sufficient to decide whether it should be authorized to access secret information. For example in Figure 3, suppose component ”A” authenticates ”B” against a hash of its code. Since ”B”’s hash is fixed in ”A”’s code, a malicious host cannot substitute a malicious version of ”B”. However, if ”B” also communicates with component ”C” (whose hash is not hardcoded by ”B”), then if ”B”’s hosts provides ”B” a malicious hash for ”C”, ”B” may leak ”A”’s messages to ”C”. Even though ”A” authenticated ”B”’s code, it could not authenticate which components ”B” trusted.

This scenario is a violation of the decentralized security principle [21], a design principle stating that one’s security should never depend on components of the system that one does not trust. Since ”A”’s code did not express trust in ”C”, it should not be possible for ”C” to disclose ”A”’s secrets.

Fig. 3: Allowing hosts to authorize enclaves independently could permit flows to malicious enclaves.

Since it is only possible to hardcode component hashes that exist at compile time, most TEE applications with multiple components face some version of these problems with mutual authentication and transitive trust. Like many challenges in security, this one is easily solved if we introduce a trusted third party; for example, if components only accept expected hash values if they are signed by the developer. While this approach is an improvement since it shifts trust from a component’s host to a trusted third party (e.g., the component’s developer), it undermines the security of the system since the third party has the power to arbitrarily change the components of the system. Note that because information may propagate through multiple components as in Figure 3, the party must be trusted by all components in the system. In decentralized systems, it may be that no universally-trusted entity exists.222Strictly speaking, all entities in a system secured by TEEs must trust the TEE manufacturer. In this paper, we assume that entities trust the TEE and its manufacturer for authentication, but not authorization. That is, entities accept an attestation as proof of what component is running in a remote TEE, but do not rely on the manufacturer to determine which components are trustworthy. A malicious manufacturer (or catastrophic flaw in its implementation) could subvert the attestation process to authenticate unauthorized TEEs as authorized ones, but we consider such attacks outside the scope of this paper. To ensure the end-to-end security of distributed and decentralized applications, components need more flexible and expressive mechanisms for authentication.

Another challenge for decentralized TEE applications arises when components are replicated for scalability. For example, serverless applications such as those built on AWS Lambda,333 Google Cloud Functions,444 or Azure Functions,555 reduce resource costs by factoring their program logic into stateless components that interact simply with persistent data storage. If demand for the component suddenly increases, new replicas are launched to meet the demand. If demand drops off, replicas may be killed to reclaim their resources. Therefore, to take full advantage of serverless platforms, components need to have relatively low startup costs and be able to process requests from any client, even if a different replica previously processed requests from the same client.

Remote attestation composes poorly with serverless design in part because an attestation only authenticates a specific replica. Whenever a client is presented with a new replica, the attestation protocol must be repeated and authenticated. Repeated attestations can introduce significant latency. For example, the standard Intel SGX remote attestation protocol requires a client and server to exchange at least five messages, and additionally requires communication with the Intel Attestation Service to verify the attestation report  [22, 5].

Distributed TEE applications frequently amortize this cost by agreeing on some cheaper, ephemeral means of authenticating future communication such as message authentication codes (MACs) based on a shared key. Unfortunately, since the cost of remote attestation is so high, the cost is fully amortized only for long sessions with many requests. Since most serverless-style applications frequently spawn new replicas and have relatively short sessions, reducing the overhead of authenticating new replicas could significantly improve the performance of applications that use remote attestation.

Finally, the software development lifecycle also presents challenges for decentralized TEE applications. Remote attestation allows developers (and indirectly, users) to specify how specific binaries may interact, but code changes frequently over the lifetime of an application. Updating one distributed component should rarely result in downtime for the entire system. Therefore, developers need a mechanism to securely authorize new components and revoke old components even after a system is deployed.

To address these challenges we have developed the Decent Application Platform, a framework for building secure decentralized applications using TEEs. Instead of forcing developers to build ad-hoc authorization mechanisms on top of remote attestions, Decent developers refer to the components their application depends on using a high-level service name. At deployment, Decent nodes specify an authorization list that defines one or more components that are authorized to implement each service. At runtime, the Decent platform authenticates each remote component and ensures it is authorized to perform the desired service.

Decent ensures that malicious hosts cannot compromise the confidentiality or integrity of a Decent application by replacing components the application depends on. Since Decent components execute within TEEs and are authenticated using remote attestation, the confidentiality and integrity of the application does not depend on the trustworthiness of the host or its operator: any host may provide the service. We have implemented two Decent applications to evaluate the expressiveness and performance of our design. DecentRide, a decentralized ride-sharing application, and DecentHT, a simple distributed hash table.

We evaluated DecentHT’s on the YCSB [11] benchmark and compared the overhead of Decent’s authorization mechanisms to a traditional remote attestation approach. Our results demonstrate that using Decent improves throughput for shorter sessions by as much as 7.5x and latency by as much as 3.67x.

The source code of Decent SDK, DecentRide, and DecentHT including the code for benchmark have been released on GitHub.666The link is omitted for anonymity.

The rest of this paper is organized as follows: Section 2 motivates the design of Decent using our decentralized ride-sharing example. Section 3 gives a high-level overview and the design in depth of the Decent Application Platform. Section 4 outlines our implementation, and Section 5 and Section 6 evaluate the expressiveness and performance of our example Decent applications. Moreover, Section 7 provides discussions on existing works that are related to Decent. Finally, Section 8 concludes the paper.

2 Motivation: DecentRide

To motivate the design of the Decent framework, we will use our decentralized ridesharing application as a running example. Ridesharing services such as Uber and Lyft match riders to drivers who pick up one or more passengers and drive them to their desired locations.

Current ridesharing applications are highly centralized. All aspects of the system are controlled by the ridesharing company: setting prices, suggesting routes, matching riders and drivers, and processing payments. Moreover, all the data associated with these tasks is accessible to the ridesharing companies, raising concerns for passengers who may wish their travel patterns and other personal data to be kept confidential. The dominance of current ridesharing companies also gives drivers few alternatives when prices or policies are disadvantageous to the drivers’ interests.

Fig. 4: DecentRide, a decentralized ridesharing service.

A decentralized ridesharing application could address some of these concerns by letting drivers, passengers, and service providers self-organize, but designing such an application has several security challenges. Figure 4 illustrates the interactions between components of DecentRide, a ridesharing service loosely based on Uber’s microservice-based design.777

In a decentralized application, these components may be hosted by multiple entities, some of which may be untrustworthy. For example, a malicious entity hosting the Trip Planner component could learn the locations and routes of drivers and passengers, and a malicious Billing Service component could manipulate prices.

Trusted execution environments like Intel SGX are a useful tool for implementing DecentRide since remote hosts can establish the authenticity of a remote component via remote attestation, and communicate over authenticated, encrypted channels that are inaccessible to the component’s host. Unfortunately, using TEEs introduces additional challenges. First, since components may be hosted by untrustworthy entities, they must mutually authenticate each other, leading to the circular dependencies described in Section 1.

Resolving these dependencies requires some authorization data to be provided by the host at load time, so it is critical that an untrustworthy host cannot manipulate this data to subvert the security of data processed by the application. Furthermore, since some DecentRide components do not directly connect to each other, they cannot be directly authenticated and authorized by all components. Therefore, avoiding transitive trust attacks as illustrated in Figure 3 is critical for security. For example, since the Billing Service only interacts with the Trip Planner, the host of the Trip Planner might provide authorization data that would allow a malicious Billing Service to manipulate prices.

Second, updating components in a decentralized application is challenging since it is difficult to distinguish legitimate updates from malicious ones. As in the case of circular dependencies, components that do not exist at compile time cannot be authorized by the code they contain. Therefore additional data must be provided to authorize new components. The specific authorization process may be application specific, but any runtime process that authorizes new code (or revokes the authorization of existing code) should itself be authorized by the entities in the system whose security is at stake; otherwise, the process might be used to introduce malicious components that undermine security.

Finally, because of the high cost of authenticating components using remote attestation, most TEE applications establish an ephemeral session key during authentication, amortizing the cost over the lifetime of the session. Unfortunately, since each component in a microservice-based designs like DecentRide’s may be replicated in response to demand (and stopped when demand drops), the lifetime of each component may be relatively short. These shorter lifetimes make remote attestation a significant performance bottleneck, especially for Intel SGX TEEs, where one must contact the Intel Attestation Service [5] to determine whether an attestation is authentic.

3 Decent System Overview

Threat model

Any entity may host a Decent node. Malicious hosts may attempt to access and manipulate all inputs and outputs to Decent components, including local memory and storage, messages between components, and configuration data such as authorization lists. Communication channel is not limited to any type of network connections, since Decent components estabilish TLS connection between each other. We assume the security of the TEE mechanism for providing confidential computation within the TEE that cannot be directly subverted by malicious hosts, and that hosts are unable to compromise cryptographic mechanisms such as public-key encryption or digital signatures with non-negligible probability. We also assume users and hosts will get the digest of the enclave programs they trust as how they did before.

Each Decent component is part of the trusted computing base (TCB). Decent does not prevent vulnerabilities in these components, but rather provides a mechanism for all entities involved in a decentralized application to agree upon which components should be part of the TCB, and to verify that only authorized components receive access to protected data. To receive the security guarantees of the Decent platform, applications must use the Decent API to establish and authorize all remote connections between application components. We also assume that honest hosts and users only authorize connections to components that use the Decent API.888Including components in an authorization list that make non-Decent connections is supported, but the behavior of these components should be carefully considered before authorization since they could subvert application security.

The Decent platform currently only provides confidentiality and integrity guarantees. Trusted execution environments alone cannot provide availability guarantees since untrustworthy hosts can always suppress messages sent to or from the TEE, or simply shutdown the TEE altogether. We plan to extend the Decent platform with additional mechanisms for ensuring an application’s services and data remain available in future work.

Furthermore, if the communication patterns of a Decent component varies based on secret data, an attacker may be able to infer secret information by observing those patterns, even if the contents of messages are encrypted. Preventing such leaks precisely and efficiently requires tracking the control-flow dependencies that lead to behavior that is observable to an attacker (such as a message send). We currently assume that Decent components do not exhibit such leaks, prevent them via static or dynamic information flow control (IFC) mechanisms (e.g., [21] or [27]), or compensate for them with oblivious computing techniques (e.g., [20, 32]).

To support decentralized applications such as DecentRide, Decent extends existing approaches to remote attestation with new mechanisms for authorizing application components. Each Decent component creates a unique key pair that is not only bound to a specific, authentic Intel SGX enclave (ensuring the private key is inaccessible to any entity outside of that enclave), but is also bound to a specific Authorized List which contains a list of service names and the components that are authorized to provide them. The Authorized List is provided by untrusted hosts of each component, so it could potentially contain malicious entries. Therefore, Decent components only authorize connections to components with matching Authorized Lists. Hosts cannot modify the Authorized List without launching a new component (and thus attesting with a new key pair), so each Decent component is identified by a public key that is bound to a specific TEE and a specific Authorized List. Decent authorization is decentralized in the sense that host may include arbitrary entities for the Authorized List of components they host, but these entries constrain which components are willing to use their components. Thus, if a group of hosts colluded to include a malicious component, any client or legitimate enclave compoenet attempting to connect would see the unknown component in the Authorized List, and refuse the connection.

By only connecting to components with matching Authorized Lists over secure, authenticated channels, users of Decent applications have a complete picture of the components that may access and process their sensitive data, even if the hosts of those components may change dynamically in response to demand. Since any entity may host a Decent component, applications may use any desired method for discovering new hosts and load-balancing requests across them. Because these services cannot affect the confidentiality or integrity of the application, they need only be trusted with the application’s availability.

3.1 Authenticating with self-attestations

The first time a Decent component is executed, it creates a key pair and initiates the remote attestation protocol to bind the public key and Authorized List to the code running inside the enclave. We call this process “self-attestation” because the host of the component plays the role of the remote verifier in the protocol. The goal of self-attestation is not to authenticate the component to the host (since the host was the one to launch it), but instead to create a certificate verifiable by a third party.

Fig. 5: Self Attestation Process

Figure 5 illustrates the self attestation process. First, the Decent component produces a remote attestation report signed by the EPID [8] group signing key provisioned by Intel to the CPU. In addition to the platform information and digest of the component’s code that is usually included in the report, the Decent component also includes the fingerprint of the public key it generated. The host sends this report to the Intel Attestation Service (IAS) for verification. If the signature is valid, the IAS returns a signed response, verifiable by a well-known public key.

Upon receiving and verifying the IAS report, the Decent component creates a signed X.509 certificate including the RA report, its public key, and the Authorized List provided by the host. By verifying the RA report using Intel’s public key, third parties can confirm that the public key was created by a specific Decent component in an authentic SGX enclave. By verifying the signature on the certificate, third parties can confirm the authenticity of the Authorized List.

Self-attestation certificates can also be used to verify outputs of a Decent application. For example, the DecentRide Payment service could provide digital receipts that verify a user’s payment for a particular trip. By signing the receipt and attaching its self-attestation certificate, any third party can verify the contents of the receipt was produced by a legitimate instance of the DecentRide app containing no unauthorized components, even if some of those components are no longer running.

Self-attestation benefits applications like DecentRide by reducing the latency overhead of component authentication. Rather than performing remote attestation directly during the authentication process, Decent components authenticate each other using self-attestation certificates created at load time. The traditional approach of establishing an ephemeral key during remote attestation scales poorly for applications like DecentRide where component-to-component sessions may be short lived. Because self-attestation certificates are reusable across sessions, even applications with short sessions scale well in Decent.

A potential downside of self-attestation is that it increases the amount of time it takes for revocations to take effect. If a CPU is added to Intel’s revocation list after a self-attestation certificate is created, the component may continue to authenticate itself as long as the certificate is valid. However, setting short lifetimes for these certificates and periodically refreshing attestation reports in parallel can achieve similar revocation guarantees when desired. We discuss revocation in more detail in Section 3.4.

3.2 Authorizing Decent components

Code Digest Service Name
dff1…8e41 BillingService
3fb5…cc46 PaymentService
6233…0f6d TripMatcher
717e…5c1b TripMatcher
Fig. 6: Example Authorized List for DecentRide

Components in a decentralized application cannot mutually authenticate using remote attestation without some additional mechanism that specifies which components are authorized. Instead of using a trusted third party for managing component authorization, Decent allows untrusted hosts to unilaterally specify authorized components when launching a component.

Figure 6 illustrates an example Authorized List for the DecentRide example. Each entry in the Authorized List maps the hash of a component’s code to the service name it is authorized to implement. Note that a service may be provided by multiple components: for example, the TripMatcher service may be provided by two components. The initial connection to a remote component is established outside the TEE and the connection information is passed into the enclave to authenticate the component and establish a secure channel. Code running inside the TEE is agnostic to the specific host providing the service—only the service name is necessary for establishing a secure connection to the remote component.

Components authenticate themselves by submitting a self-attestation certificate for verification. A component wishing to authorize a remote component first verifies the signatures of the certificate (including the attestation report signed by Intel), then compares the contents of the remote component’s Authorized List to its own version. If the Authorized Lists match, then the code digest of the remote component (contained in the certificate) is compared to the entries in the local Authorized List to determine if it is authorized to perform the expected service.

Authorized Lists are fixed at load time and cannot be modified through the Decent API. Since the Authorized Lists of all connected components must match exactly, each instance of a Decent application is represented by a single Authorized List. Malicious components, of course, are not restricted to using the Decent API at all. However, in order to attack an application instance a malicious component must first be authorized to connect to it. For example, consider a malicious version of the Driver Management component. Figure 7 illustrates a malicious host attempting to introduce this component by launching an authorized Payment Service with the malicious component on its Authorized List. Even though the Payment Service is authorized, the Trip Matcher rejects connections to and from this component since its Authorized List list differs.

Fig. 7: Prevent malicious components by comparing Authorized Lists

Therefore, even though hosts may define Authorized Lists arbitrarily, they cannot be used to introduce unauthorized components into a target application instance: the current components will reject any component whose Authorized List deviates from the instance’s list, even if the component is otherwise authorized. Clients may examine the Authorized List of any component to determine whether they should trust a specific application instance with their data. Thus even if a host (or group of hosts) creates a malicious instance of a Decent application that includes malicious components, this instance will never be entrusted with sensitive data.

3.3 Dynamic component authorization

Requiring all Authorized Lists in an application instance to match prevents unauthorized components from connecting to the instance, but it also prevents new components from being authorized dynamically. Applications may wish to dynamically authorize components for a number of reasons, but a common reason is to update components to add features, improve performance, or fix bugs. Since components are authorized based on a digest of their code, these new components will not be allowed to connect to existing application instances. To authorize new components, all existing components must be restarted with a new Authorized List.

Requiring full system restarts for component updates goes against the typical microservice-based design workflow where components are frequently and independently updated, and multiple versions of a component may co-exist at runtime. To avoid the downtime associated with such restarts, Decent distinguishes two special roles that enable dynamic authorization and revocation: verifiers and revokers.

A verifier is a Decent component meaning that it is also an enclave program, which is permitted to authorize new components by an application instance. Comparing to trusted third parties, trusting a verifier is expressing trust only in the specific code running in the enclave, and the enclave platform ensures that the enclave’s behavior cannot deviate from that code.

Like any other Decent component, in order to join an application instance, the verifier must be listed as a verifier on the Authorized Lists of the components in the instance. In other words, all components must agree on which verifiers are authorized to make dynamic authorization decisions. By defining multiple verifier service names, applications can designate which verifiers are authorized for which services. For instance, the DecentRide application could define a BillingServiceVerifier name for authorizing BillingService components, and a TripMatcherVerifier for authorizing TripMatcher components.

Verifiers authorize new components by signing the component’s self-attestation certificate with their own private signing key. A component is authorized to connect to an application instance if (a) it appears in the Authorized List for the expected service or (b) its self-attestation certificate is signed by a verifier who appears in the Authorized List for the expected service verifier. In both cases the Authorized List of the new component must match the application instance’s. In the latter case, the verifier’s self-attestation certificate must also contain a matching Authorized List.

Fig. 8: Procedures for the Example Verifier

Verifiers may use any desired mechanism to decide whether a component should be authorized. One approach is for the verifier to collect signed approvals from an application’s “stakeholders,” e.g., the entities whose data security may be affected by the authorization. Figure 8 illustrates this workflow. The initial application instance contains Enclave A, Enclave B, Enclave C, and Verifier, which contains a hardcoded list of the public keys of stakeholders whose approval is required for new authorizations.

When New Enclave is created, stakeholders that wish to authorize New Enclave create and sign New Enclave’s code digest. These approvals may be submitted to Verifier in a number of ways. They could be sent by the stakeholders directly, or they could be posted publicly (or distributed with the component) and submitted by the host of New Enclave as a batch. In either case, New Enclave authenticates itself to Verifier using its self-attestation certificate. If Verifier has received the necessary approvals, it responds by signing New Enclave’s certificate. Finally, New Enclave joins the application instance by presenting its self-attestation certificate, Verifier’s signature, and Verifier’s certificate to Enclave B. Enclave B authorizes the new component by verifying the self-attestation certificates and Verifier’s signature, and by checking that Verifier is on its Authorized List.

Other dynamic authorization approaches could remove the need for external entities to explicitly authorize new components. For example, Fabric [21] permits mobile code to be downloaded from untrusted hosts, formally verified for information-flow security, and linked into a distributed application at runtime. A similar approach could be adapted for Decent verifiers. Each new component’s source code would be checked by the verifier against a formal specification associated with the service it claims to implement. If the source is successfully verified, the verifier compiles the source to machine code, calculates the hash of the new component and issues an signed approval. Any enclave presenting a valid self-attestation certificate for the generated component will be signed by the verifier, without the need for additional approvals.

3.4 Revocation

Decent components may have their self-attestation certificates revoked in one of two ways. First, the Intel Attestation Service maintains a number of certificate revocation lists (CRLs) it uses to determine whether an SGX platform (the chip, the credentials provisioned to it, or the platform software itself) have been revoked. Since remote attestation reports are signed by a group signature to protect privacy, only Intel is able to distinguish revoked platforms within a group. A component running on a host whose platform has been revoked will be unable to refresh its self-attestation certificate with the IAS server. Therefore it is prudent to set self-attestation certificates to expire at reasonable intervals to force periodic refreshing.999While a malicious host may manipulate the operating system clock, the SGX platform provides a trusted interval timer that can be used as the basis of a mechanism to measure certificate lifetimes and force refreshes. We leave the design and implementation of such a mechanism to future work.

Dynamic component revocation

If authorized components are discovered to have latent vulnerabilities, or are incompatible with new updates, these components should no longer be permitted to connect to a Decent application instance. However, dynamically revoking component authorizations presents many of the same challenges as dynamic authorization. Forcing full system restarts to modify Authorized Lists leads to increased downtime, and may delay when revocations take effect. Furthermore, modifying Authorized Lists does not address revocation for dynamically authorized components.

Decent revokes authorized components using Component Revocation Lists (CoRLs). A CoRL is similar to Certificate Revocation Lists [12] maintained by a Certificate Authority, but instead of revoking a specific self-attestation certificate, a CoRL entry is used to revoke any self-attestation certificate generated by a specific Decent component.

Decent components called revokers maintain the CoRLs associated with an application instance. Like verifiers, revokers must appear on the Authorized Lists of all components in the application instance, or must themselves be authorized by a verifier. The mechanism by which a revoker adds new entries to its CoRL is application specific, but our Decent prototype does not yet include support for revoker components.

Fig. 9: Procedures for the Example Revoker

Figure 9 illustrates a revocation workflow based on a group of stakeholders, similar to the verifier example of Figure 8. Stakeholders submit revocation requests to one or more revoker enclaves containing the hash of the component whose authorization is to be revoked. Once a threshold of revocation requests are received, the component’s hash is added to the CoRL. Enclaves in the application instance periodically request updates to the current CoRLs from designated revokers. To prevent revocations from being suppressed by malicious hosts, enclaves refuse to proceed if they do not receive a response from the revoker.

In some scenarios, it may be possible to revoke a component without the intervention of an external entity. If evidence that a component is compromised is mechanically verifiable, as in the case of conflicting protocol messages or a leaked cryptographic key, then this logic can be included in the revoker component to automatically add entries to the CoRL.

Revokers may revoke the authority of any Decent app or verifier. Revoking the authority of a server or revoker component dynamically is problematic. Server components are designed to be small and rarely updated. Many components in an application instance will likely share the same server component, so revoking the server would invalidate the Decent app certificates of many components, and these components would be unable to re-attest and rejoin the instance until an authorized Server component is launched by their host. Revoking a verifier can similarly cause many components’ self-attestation certificates to be rejected, but unlike the server scenario, these components may rejoin as long as they are able to locate some other authorized verifier that belongs to the instance.

Decent prohibits the revocation of revoker components for two reasons. First, the desired effect of revocation is unclear. For example, should components on a revoked revoker’s CoRL be allowed to rejoin the instance? Even if some entries on the CoRL are erroneous, permitting all components to rejoin could undermine the security of the instance. Second, If two different revokers place each other on their CoRL, which revocation should take precedence is unclear.

Finally, revokers may be dynamically authorized by verifiers, but these verifiers must be treated specially. Since revoking the authority of a verifier of revokers would lead to the same issues that accompany revoking a revoker directly, the verifiers of revokers cannot have their authority revoked dynamically.

3.5 Accessing and Migrating Sealed Data

Persistent data stored by an enclave must be encrypted with keys that the enclave maintains access to across reboots of the enclave process. Embedding secret keys in enclave code is obviously insecure, and keys that are stored in the (encrypted) enclave memory will be lost if the process exits. Intel SGX provides two key-derivation schemes for creating seal keys that encrypt data for persistent storage outside of the enclave. One scheme, MRSIGNER, uses the signer (typically the author) of the enclave to derive seal keys, meaning that any enclave binary signed by the same key may decrypt sealed data. The other scheme, MRENCLAVE, uses the enclave’s code hash to derive keys, meaning that only that enclave may decrypt sealed data.

Neither of these key-derivation schemes alone are appropriate for Decent applications. The author of an enclave has no special authority in Decent. Entities do not place trust in the authors of components, only in the code itself. However, allowing a Decent component to access sealed data could be insecure if the component belongs to a different application instance than the one that sealed the data. Therefore, the Decent SDK uses a HMAC-based key-derivation function [18] (HKDF) to derive a key from the MRENCLAVE-derived key that binds the key to a specific Authorized List.

Decent’s HKDF scheme prevents malicious hosts from using legitimate components to leak sealed data to a malicious components (since the Authorized List would be different), and malicious components from directly deriving another component’s seal key (since the enclave’s hash would be different). However, it also implies that sealed data must be explicitly migrated from one application instance to another if the Authorized List changes or a component is upgraded.

One approach to data migration is to implement a migration API by which a new component can request sealed data from an existing component before it is replaced. For example, if a new component is authorized by the verifier, it can contact a specified component from which to migrate data and seal the retrieved data under its own key. In some scenarios, however, it may be unreasonable to use the existing component to migrate sealed data. For example, if the Authorized List is changed, or a component is about to be revoked because of a vulnerability, delaying revocation to migrate data to a replacement component could subject the application instance to exploitation. Guarding against sudden revocation of a component with a large store of sealed data requires sealing the data under a key that can be provisioned to “recovery components” that can access and migrate data securely in the event the component’s authority is revoked.

Note that TEE mechanisms like Intel SGX cannot on their own be used to guarantee recovery of sealed data. A malicious host may deny access to sealed data at any time. We are currently investigating mechanisms to integrate into the Decent framework that would support availability guarantees in additional to the current confidentiality and integrity guarantees.

4 Implementation

We have implemented a cross-platform prototype of our design using Intel SGX for Linux and Windows. Our prototype consists of about 20k lines of C++ (12k excluding header files) and uses Intel SGX SDK version, and mbedTLS version 2.16.0. While some of our design decisions are informed by the constraints of the SGX platform, our high-level design is applicable to any TEE platform that supports remote attestation and secure memory. Our prototype separates the concern of generating self-attestation certificates from the application logic of each component. Self-attestation is performed by a Decent Server component whereas all component code resides in a Decent App component. Each Decent node runs a single Decent Server component, regardless of how many components (or application instances) it hosts. Therefore only one self-attestation certificate is necessary per node.

Fig. 10: Local Decent App authentication

Figure 10 illustrates the authentication process between a Decent Server and an App component. Once the Decent Server has successfully created its self-attestation certificate, a Decent App may request an App Certificate. The Decent App must then locally attest to the Decent Server, which provides similar guarantees to remote attestation but does not require verification by the IAS since the enclaves reside on the same SGX CPU. Using a shared key derived during the local attestation process, the Decent App sends its Authorized List and a Certificate Signing Request (CSR) containing the public key of the Decent App component. The Decent Server then returns a signed Decent App certificate containing the App’s public key, the digest of the component’s code, and the App’s Authorized List. The Decent App subsequently presents this certificate chain to remote components for authentication.

Factoring out the self-attestation code keeps the self-attestation protocol simple, easier to understand and audit, and reduces the size of application components. Since the protected memory available to SGX enclaves is severely limited (128MB with less than 95MB is usable), keeping the size of components small is critical for performance. The Decent Server is about 3k (non-header) lines of C. Including this code in the DecentHT application would increase the lines of code by .

Furthermore, bootstrapping the authentication of components using a single self-attestation certificate for the Decent Server reduces the overhead of authentication on hosts that run multiple Decent App components. Since each Decent App component is running in a separate enclave, they cannot access the memory of the Decent Server or other components, so even a malicious Decent App cannot tamper with the authentication process of other components.

4.1 Decent App Handshake

Fig. 11: Decent handshake workflow

Decent App components communicate over TLS connections that are authenticated with their App certificates. The Decent handshake extends the usual TLS handshake where both sides authenticate with X.509 certificates.

Figure 11 illustrates the handshake procedure. First, Enclave A and B exchange certificate chains, whose root is the IAS report, which is included in the Decent Server’s certificate, self-signed by the Decent Server enclave itself. The certificate of the Decent App or verifier is signed by the Decent Server enclave. Additionally, if the Decent App is verified by the verifier, its certificate is signed by the verifier enclave. The mbedTLS library verifies the certificate chain and then executes a callback function that performs additional checks on their contents. First, the self-attestation report signature is verified with Intel’s public Report Key. Next, the local Authorized List is consulted to ensure the Decent Server’s hash appears under the “DecentServer” service name. The Decent App’s hash must either appear under the expected component service name in the Authorized List, or its App certificate must be signed by a Verifier whose App certificate is included (and verified) in the certificate chain, and whose hash appears under the expected verifier service name. During this process, the revocation list must be consulted to ensure all hashes—Servers, Apps, and Verifiers—are still valid and have not been revoked.101010Our current prototype does not yet support revocation. Finally, the Authorized Lists included in each App’s certificate are compared to the local Authorized List to ensure they match. If all checks are successful, the connection is accepted, otherwise it is rejected.

4.2 Using the Decent SDK

Since any code running outside of an enclave can be subverted by a malicious host, creating secure communication channels between remote components is the core service a decentralized TEE application framework must provide. The Decent SDK provides a high-level API for establishing secure channels between components that greatly simplifies authentication and authorization of remote application components while preserving fine-grained control over which components are authorized to perform specific services.

System calls such as those that create, read, or write network connections cannot be executed within an SGX enclave. Therefore untrusted code must perform all network operations. The standard approach [17] for establishing a secure channel with an enclave is for untrusted code to first create (or accept) a TCP connection to (or from) the remote host, and then act as a proxy between the enclave and the network. Bytes are passed into a send buffer by the enclave and written to the network, and bytes are read off the network into a receive buffer and returned to the enclave.

[ linenos=true, breaklines=true, tabsize=4, mathescape, fontsize=] c++figures/SampleCode-TlsWithName.cpp

Fig. 12: Setting Up A TLS Connection to Decent App with Authorized List

Inside the enclave, the code treats the send and receive buffers as it would a traditional network connection. By reading and writing TLS protocol messages, it may establish an encrypted and authenticated channel to a remote host. This channel is protected from the host of the enclave since the secret keys used to establish the channel are inaccessible to the host.

Figure 12 shows a fragment of code from the Payment Service component in DecentRide that handles incoming requests from the Trip Matcher component. The void* pointer cnt_ptr points to a TCP connection created by the untrusted code and passed into the enclave. The authorization data of the component is retrieved on line 5, and wrapper class for the TCP connection is instantiated on line 7. Lines 10-15 create an object that configures how the connection should be authenticated and authorized. The authorization data state is passed in to provide the TLS library with the component’s key pair, App certificate chain, and Authorized List. The mode ServerVerifyPeer indicates that the Payment Service is expecting an incoming request (hence Server) and should request a certificate from the remote component (hence VerifyPeer). Line 14 specifies that the expected service name of the remote component is ”TripMatcher”, thus the service name ”TripMatcher” must be listed under the component’s hash entry in the Authorized List. On line 15, ”GetSessionTicketMgr()” retrieves a reference to a TLS session ticket manager that helps resume sessions to avoid re-negotiating the TLS handshake unnecessarily. Lines 18 and 19 establish the TLS channel using the connection wrapper and the configuration object, and lines 20 and 21 use the channel to receive and send data with the remote component.

[ linenos=true, breaklines=true, tabsize=4, fontsize=, mathescape, firstnumber=10 ] c++figures/SampleCode-TlsWithVrfy.cpp

Fig. 13: Setting Up A TLS Connection to Decent App with Verifiers

Permitting TripMatcher components to be authorized using a Verifier component only requires a few modifications to Lines 10-16 of the Payment Service code above. In Figure 13, we instead create a shared pointer to a ”TlsConfigWithVerifier” object, whose constructor accepts an additional service name, ”TripMatcherVerifier” for the verifier service name permitted to authorize ”TripMatcher” components dynamically. This configuration requires that any verifier of a ”TripMatcher” component be listed under the ”TripMatcherVerifier” service name in the Authorized List.

5 Example Decent Applications

5.1 DecentRide

We used our prototype to implement two Decent applications. The first application is an implementation of DecentRide, discussed in Section 2. DecentRide components are hosted by a distributed network of mutually-distrustful nodes. Because these components run inside SGX enclaves, confidential information like the location and routes of drivers and passengers is kept private, and critical computations like route and pricing calculation cannot be manipulated by the hosts.

Each DecentRide component (circles in Figure 4) implements a microservice—a small, targeted service that is loosely coupled with the other components by a simple application protocol. With the exception of the TripMatcher, DecentRide components maintain no local state. This design makes it easy to launch (or halt) component replicas on demand and balance request load among hosts. DecentRide hosts are incentivized to host components by compensation they receive (calculated by the Billing Service) from payments for trips (processed by the Payment Service). Passengers and drivers register and authenticate to the Passenger Management and Driver Manager services, and receive credentials that are presented to (and verified by) the DecentRide components they interact with. All user data provided to the management services, including their contact and payment information, is encrypted and signed with keys derived from a component’s seal key and the Authorized List as described in Section 3.2. Thus hosts have no access to the user data they store, even if they launch malicious DecentRide instances.

When a passenger wishes to find a ride on DecentRide, they contact a Trip Planner service which finds a path between their current location and the desired destination. The Trip Planner submits the proposed route to the Billing Service to get a total price for the trip, including fees for the hosts operating DecentRide components. This price quote is signed by the Billing Service and returned to the passenger (via the Trip Planner) along with the Billing Service’s certificate chain, which the passenger and Trip Matcher uses to verify the authenticity of the price quote.

Drivers advertise their availability by sending their location data to a Trip Matcher, which replies with a list of nearby passengers and their desired destinations. Once a driver selects a passenger, their contact information will be exchanged, and the passenger will receive the current location of the driver. At the destination, both driver and passenger must confirm to the Trip Matcher that the trip completed successfully, and only then does the Trip Matcher initiate payment for the trip with the Payment Service. The Payment Service obtains payment information from the management services and submits the transaction to the relevant financial entities.

To prevent abuse such as fake trip requests designed to probe for driver locations, or fake driver locations to probe for passenger locations and destinations, driver and passenger interactions with the Trip Matcher are logged to their respective management services. The Trip Matcher will not proceed with a driver or passenger request until it receives confirmation from the service that the logged action was successfully received, ensuring that hosts of the Trip Matcher and management services cannot collude with malicious drivers or passengers to suppress these logging messages.

5.2 DecentHT

To complement to DecentRide, we have implemented DecentHT, a simple distributed hashtable based on Chord [28]. Running each node within an enclave lets us store confidential information at untrusted hosts and control access to that information using Decent Authorized Lists. Sealed data stored in the DecentHT can be accessed based on a consistent hash function applied to the desired key, just as in the original Chord system.

A non-enclave alternative to DecentHT could store encrypted values that were inaccessible to their hosts, but hosts would still be able to observe which keys are accessed. To encrypt the keys, deterministic encryption would be required, which would still reveal key access patterns if not the key values. Furthermore, controlling access to the decryption keys introduces extra complexity. DecentHT nodes might still learn some information from analyzing communication patterns between nodes, but at a much slower rate when there is a high ratio of keys to nodes.

DecentHT nodes encrypt their data using their own sealing keys, and only provide access to authorized entities. It requires no additional key management beyond the Decent authentication mechanisms. Executing the Chord protocol logic within the enclave rules out attacks based on manipulating protocol messages or routing attacks since even malicious hosts process messages with trusted code. Host may still, of course, suppress incoming or outgoing messages.

DecentHT derives each node’s identifier, which determines which keys it is responsible for on the consistent hash ring, using the Decent seal key. This approach prevents many Sybil attacks [14] since every DecentHT component launched with the same Authorized List on the same SGX CPU will receive the same identifier. Components launched with different Authorized Lists will be unable to connect to the same DecentHT instance.

Fig. 14: DecentHT: a Decent implementation of Chord [28].

Figure 14 illustrates a DecentHT network used by a DecentRide instance to store passenger and driver data used by the management services. Drivers and passengers interact with the management services, which may cause them to create, fetch, or modify data stored in DecentHT. As in Chord, each node in a -node system stores routing information for roughly other peers so that incoming requests for keys not stored remotely can be routed to the appropriate node. When the component responsible for the key is finally reached, the component loads the associated (sealed) data into the enclave and sends it to the requesting application.

6 Performance Evaluation

6.1 Experiment Setup

We evaluate the overhead of Decent authentication and authorization by comparing the performance of DecentHT on the YCSB benchmark [11] to an implementation that uses a remote attestation approach as suggested by the Intel SGX SDK documentation. Since the SGX RA Only approach does not support mutual authentication of the DecentHT components, we omit the comparison of component hashes to an expected value in this version. In our results, we refer to the Decent implementation as Decent RA and the SGX SDK implementation as SGX RA Only.

For additional context, we also evaluate the performance of two non-enclave implementations to examine the baseline performance of the system without the overhead of authentication or SGX operations. In one version, the DecentHT code executes outside of the enclave and communicates over TLS channels without exchanging and verifying certificates. In the second version, we additionally encrypt the stored records with a symmetric key, similar to how the enclave versions seal the records for storage outside of the enclave’s memory. We refer to these implementations as TLS Only and TLS+Sealing, respectively.

Our experimental setup is similar to Figure 14, but instead of DecentRide components, we created a small Decent App that exposes Java bindings for the YCSB benchmark to invoke. We used Workload B, a “mostly read” workload with reads and writes, with uniform request distributions for all our experiments below. A read operation consists of one lookup request to determine the node responsible for storing the desired record, followed by a request to fetch the record. Write operations are one lookup request followed by an update request. Each DecentHT node loads approximately 3,000 records. To ensure a relatively even distribution of records across nodes, we disabled DecentHT’s Sybil resistance and explicitly specified node ID’s that were evenly spaced. Each test measures performance for 60 seconds after a 60-second warm up phase. We repeat each test three times and report the median of measurements as points on the graph, while, minimum and maximum values as error bars, which are not distinguishable in our graphs.

DecentHT nodes execute on a single 3.6 GHz Intel i7-7700 with 4 cores (8 logical) running Windows 10. The server has 16 GB of RAM, with 128MB (the maximum permitted) dedicated to SGX. Records stored at each node are sealed and stored in non-enclave memory. Each node is assigned its own logical core, with two cores reserved for the network stack and the Intel AESM service which is responsible for managing interactions between the operating system and SGX enclaves.

Clients execute on a single 3.4 GHz Intel i3-7100T with 2 cores (4 logical) running Windows 10. The client machine has 4GB of RAM and 128 MB is dedicated to SGX. The client and server machines are connected via 1-Gigabit Ethernet.

SGX requires that all thread-local memory be pre-allocated, thus the maximum number of threads used by an SGX application is fixed at runtime and is limited by the memory available to SGX. For the DecentHT nodes, we specified 18 threads for handling incoming requests from either clients or peers, 6 threads for forwarding the finger table lookup requests to other peers, and 2 threads for replying requests received from other peers. In the SGX RA Only experiments, 14 additional threads are used for requesting quotes for the enclave itself. These threads are unnecessary for the Decent experiments, but we reserve the same amount of memory in both cases. Using the additional memory to allocate more threads for the Decent experiment would further improve Decent’s performance over the SGX RA Only case.

We limited YCSB to spawn a maximum of 50 threads, the maximum number of concurrent YCSB components that fit in the available enclave memory. This did not limit the throughput of the SGX-based implementations, which were fully loaded at around 40 client threads.

The remote attestation process results in a shared secret between the enclave and the host verifying the enclave’s attestation report. In the non-Decent implementation we follow the approach suggest by examples distributed with the Intel SGX SDK to establish a secure channel using AES-GCM encryption between the distributed hash table and the application nodes. In addition, we give nonces and randomized Initial Vectors (IVs) to the encryption process for all messages, and implemented a mechanism similar to TLS session tickets 

[24] that permitted clients to resume sessions across connections without storing state on the server. The purpose of this protocol is simply to approximate the security guarantees of Decent’s authenticated channels using secrets shared during the attestation process and avoid the additional overhead of a TLS handshake.

Since our experiments generate many IAS requests in the SGX RA Only case, we use a simple IAS simulator to avoid violating the terms of use for our Intel Developer account. Instead of sending requests to the IAS, all requests are sent to our simulator which replays a single hardcoded response from the official IAS. Requestors follow the same protocol as they would for a real IAS request, but they ignore the nonce in our simulated response to allow us to replay the same response multiple times. The response times of our simulated IAS are gamma-distributed with parameters estimated from 30 IAS API response time samples collected from the IAS portal. For retrieving the current EPID signature revocation list, we measured a mean response time of 39 ms with standard deviation 24 ms. For retrieving an attestation report we measured a mean response time of 255 ms with standard deviation 70 ms.

6.2 Results

Fig. 15: Requests Per Session versus Throughput

Both the Decent RA implementation and the SGX Only implementation amortize the cost of authentication over the length of a session. Therefore the negative impact of authentication on system throughput will decrease as the average session length increases. To analyze the tradeoff between authentication overhead and session length, we ran each implementation with six server nodes on the YCSB benchmark while varying the number of requests each client made before establishing a new session. For the Decent RA implementation, each new session involved a TLS handshake and the exchange and verification of self-attestation certificate chains. For the SGX RA Only implementation, each new session required a new remote attestation. The non-enclave versions required only a TLS handshake with no certificate verification.

Figure 15 presents the results of these experiments. The non-enclave performance improvement plateaus at about 200 requests per session, with the TLS+Sealing implementation achieving a slightly lower throughput attributable to the extra overhead of encrypting and decrypting the stored records. Between 10 and 400 requests per session, Decent RA significantly outperforms the SGX RA Only implementation. For very long sessions of 800 requests, Decent RA and SGX get roughly the same throughput. Beyond 800 requests, the SGX RA Only performance approaches the TLS Only implementations, which we attribute to the additional messages required to resume a TLS session compared to our custom SGX RA session ticket scheme.

(a) Each session includes 50 requests
(b) Each session includes 200 requests
(c) Each session includes 600 requests
Fig. 16: Average Latency versus Throughput

Figure 16 plots the tradeoff between latency and throughput for sessions of length 50, 200, and 600. We gradually increased the target throughput and measured the average response time for requests. At 50 requests per session, the difference between Decent RA and SGX RA is most pronounced, with latency rapidly increasing for SGX RA as throughput exceeds 150 requests per second. At 200 request per session, the behavior has begun to converge, but Decent RA still significantly outperforms SGX RA. At 600 requests per session, however, their performance is roughly equivalent. Note that the performance of the TLS Only implementation is mostly unaffected for these session lengths.

Fig. 17: Average Latency vs. Number of Nodes

Finally, to sanity-check whether Decent RA affects the scalability of the system, we measure the average latency of requests as the number of nodes increases from three to six. In Figure 17, the results show that, as expected, latency in both implementations is largely unaffected by the small increase in nodes, but the average latency is almost four times lower for Decent RA.

7 Related Work

Several recent projects use enclaves and remote attestation to provide confidentiality and/or integrity for distributed applications, but almost none address the problem of mutual authentication. VC3 [25] allows a user to launch MapReduce workloads using cloud-hosted SGX enclaves, ensuring the confidentiality of the processed data and the integrity of the results. VC3 jobs distribute a single enclave binary to each host, avoiding any mutual authentication issues. Ryoan [15] executes untrusted modules inside multiple sandbox instances running in SGX enclaves. Ryoan authenticates the initial sandbox instances using remote attestation, but modules are identified by a public key that signs the module rather than the code itself, while the corresponding private key could be a key that is stored outside of an enclave.111111The authors claim that Ryoan could also support a software analog hash identity, but it is unclear how they would address mutual authentication. In which case, the identity could be forged by someone misuse the key.

CCF [23] uses a distributed ledger to manage which TEE components are enabled. This fills a similar role to the Authorized List in Decent. CCF nodes are more heavyweight than Decent nodes because they must participate in a Byzantine agreement protocol to process requests. Decent nodes offer fewer availability guarantees, but are only required to support the Decent attestation protocols. Coupling Decent with a BFT protocol could provide similar availability guarantees for Decent Apps.

The MesaTEE framework [29], built on Rust SGX, is a framework designed for universal secure computing using SGX. MesaTEE solves the problem of mutual authentication by relying on trusted third-party auditors [30], which only sign enclave binaries of code that passes an audit process. By contrast, Decent does not place trust in a single entity to authorize enclaves, but requires hosts in an application instance to agree upon which components are authorized to implement which services.

Prior work has investigated using SGX enclaves with secure communication protocols like TLS. Some work focuses primarily on terminating secure channels inside of enclaves. For example, Intel provides SgxSSL [1], an TLS library based on OpenSSL for establishing secure channels with an enclave. Other projects such as mbedtls-sgx [2] and wolfssl [4] provide similar functionality based on other TLS codebases. TaLoS [7] uses SGX enclaves to protect TLS session information from a compromised host. Decent uses the mbedtls library with minor modifications to establish TLS connections between enclaves.

Decent’s self-attestation process is similar to a now-common approach to authenticating TLS connections with enclaves. Knauth et al. [17] describe the process of binding a public key to an enclave that generated it using remote attestation, and include the report in a self-signed certificate. This certificate is used to establish the TLS connection so that the remote host can verify the key belongs to a genuine enclave instance. Rust SGX [31] and Open Enclave [3] offer similar support.

All the above approaches, in principle, support the option of self-attestation where the host of the enclave participates on both sides of the attestation process to aid in the creation of the certificate. None however, address the issue of mutual authentication of enclaves and therefore cannot support applications like DecentRide or DecentHT. Furthermore, our results in Section 6 quantify the performance gains of self-attestation over on-demand attestation for short sessions.

Other work has focused on supporting more general systems programming within SGX enclaves. SCONE [6], Graphene-SGX [10], and Panoply [26] support standard library functions not natively available to enclaves, such as filesystem and network I/O.

In general, these projects focus more on the local secure systems programming aspects of using SGX enclaves. SCONE does not support remote attestation. Panoply partitions applications into multiple small enclave components, and a shim library maintains a mapping between identies and code hashes that is used to verify against local attestation between components. The details of how (or whether) Panoply resolves mutual authentications between local components without a third party is not fully clear, but in any case, remote attestation between Panoply components is not supported.

Furthermore, the mechanism Panoply uses to partition components and generate the shim library requires access to all program components at compile time. Thus changing or updating any component would require recompiling and redeploying the application. Decent supports a looser coupling between components, both by permitting multiple components to implement the same service, and by allowing new components to join an application instance using verifiers.

Alternative TEE designs such as Sanctum [13] and Keystone [19] offer different approaches to remote attestation. In Sanctum, each processor holds an attestation key and a corresponding certificate signed by manufacturer. By verifying the certificate chain provided with attestation reports that culminates with the signature of the manufacturer, the report can be directly verified by the receiver without the need to contact an additional service like IAS. Keystone has a similar protocol, but offers additional roots of trust beside the manufacturer.

Intel SGX platforms recently added support for Data Center Attestation Primitives (DCAP) which allow a third party to replace the EPID protocol that requires contacting IAS. Instead, an attestation key is generated locally, and a certificate chain with Intel’s signature as a root of trust is used to verify attestation reports directly. Hence, it lacks the privacy benefits provided by EPID-based protocols, and does not fundamentally alter the trust model since Intel remains a trusted entity. However, a local DCAP service could further improve performance of Decent because of reduced latency for generating self-attestation report.

DCAP, Sanctum, and Keystone’s approaches to remote attestation reduce the overhead of remote attestation in similar ways to using self-attestation certificates. They do not however, address mutual attestation between enclaves.

Another recent feature added to the SGX SDK is Key Separation and Sharing (KSS). In KSS, six new configuration parameters are introduced. When KSS is enabled, 4 of these parameters will be added to the seal key derivation, so that the KSS-enabled enclaves with different parameters will not be able to derive the same seal key [16]. It is also possible to use KSS to enable enclaves with the same parameters to derive a shared key, but only if they are signed by the same entity. These four parameters are also included in the RA report, so their modification affects the configuration identity of the enclave in the RA report, but the enclave’s MRENCLAVE identity, based primarily on the code hash, is unaffected.

KSS can help differentiate multiple instances of the same enclave. For example, Decent could use KSS to derive different seal keys for each application instance by placing a hash of the Authorized List in the configuration parameters instead of using our HKDF approach. Furthermore, since we can specify an Authorized List in terms of each component’s MRENCLAVE identity, and bind it to the configuration id used for attestation, KSS could provide an alternate implementation path for Decent’s authorization mechanism. Nevertheless, although KSS provides additional tools for mutual authentication, it doesn’t provide a solution. Any KSS-based approach would need to address the same challenges as Decent.

8 Conclusion

In this paper, we present the Decent Application Platform, which is a framework for building secure decentralized applications. Self-attestation backed by RA provides not only a unified API for RA from different protocols, but also a scalable protocol for RA. The load-time Authorized List enables mutual attestation between different enclaves, and ensures that only authorized components can join the system. Verifier and revokers allows runtime modification to the set of authorized components. We implemented the Decent SDK on top of Intel SGX, and evaluated it with two example applications. DecentRide demonstrates the expressiveness of Decent framework when building applications with multiple enclave components, while the evaluation based on DecentHT shows that, for short sessions, Decent provides 7.5x higher throughput and 3.67x lower latency comparing to the non-Decent implementation.


  • [1] Intel software guard extensions sdk., 2019.
  • [2] mbedtls-sgx: a sgx-friendly tls stack., 2019.
  • [3] Openenclave sdk., 2019.
  • [4] Wolfssl with intel sgx., 2019.
  • [5] Ittai Anati, Shay Gueron, Simon P Johnson, and Vincent R Scarlata. Innovative Technology for CPU Based Attestation and Sealing. Technical report, Intel Corporation, 2013.
  • [6] Sergei Arnautov, Bohdan Trach, Franz Gregor, Thomas Knauth, Andre Martin, Christian Priebe, Joshua Lind, Divya Muthukumaran, Dan O’Keeffe, Mark L. Stillwell, David Goltzsche, Dave Eyers, Rüdiger Kapitza, Peter Pietzuch, and Christof Fetzer. SCONE: Secure linux containers with intel SGX. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 689–703, Savannah, GA, November 2016. USENIX Association.
  • [7] Pierre-Louis Aublin, Florian Kelbert, Dan O’Keeffe, Divya Muthukumaran, Christian Priebe, Joshua Lind, Robert Krahn, Christof Fetzer, David Eyers, and Peter Pietzuch. TaLoS: Secure and transparent TLS termination inside SGX enclaves. Imperial College London, Tech. Rep, 5, 2017.
  • [8] Ernie Brickell and Jiangtao Li. Enhanced privacy id from bilinear pairing. IACR Cryptology ePrint Archive, 2009:95, 2009.
  • [9] Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F. Wenisch, Yuval Yarom, and Raoul Strackx. Foreshadow: Extracting the keys to the intel SGX kingdom with transient out-of-order execution. In 27th USENIX Security Symposium (USENIX Security 18), page 991–1008, Baltimore, MD, August 2018. USENIX Association.
  • [10] Chia che Tsai, Donald E. Porter, and Mona Vij. Graphene-sgx: A practical library OS for unmodified applications on SGX. In 2017 USENIX Annual Technical Conference (USENIX ATC 17), pages 645–658, Santa Clara, CA, July 2017. USENIX Association.
  • [11] Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC ’10, pages 143–154, New York, NY, USA, 2010. ACM.
  • [12] D. Cooper, S. Santesson, S. Farrell, S. Boeyen, R. Housley, and W. Polk. Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile. RFC 5280 (Proposed Standard), May 2008. Updated by RFC 6818.
  • [13] Victor Costan, Ilia Lebedev, and Srinivas Devadas. Sanctum: Minimal hardware extensions for strong software isolation. In 25th USENIX Security Symposium (USENIX Security 16), pages 857–874, Austin, TX, 2016. USENIX Association.
  • [14] John R Douceur. The sybil attack. In International Workshop on Peer-to-Peer Systems, pages 251–260. Springer, 2002.
  • [15] Tyler Hunt, Zhiting Zhu, Yuanzhong Xu, Simon Peter, and Emmett Witchel. Ryoan: A distributed sandbox for untrusted computation on secret data. ACM Trans. Comput. Syst., 35(4):13:1–13:32, December 2018.
  • [16] Intel Corporation. Enclave Operation. In Intel 64 and IA-32 Architectures Software Developer’s Manual, volume 3, chapter 38, pages Vol.3D 38–1 – Vol.3D 38–16. Intel Corporation, May 2019. Order Number 325384-070US.
  • [17] Thomas Knauth, Michael Steiner, Somnath Chakrabarti, Li Lei, Cedric Xing, and Mona Vij. Integrating remote attestation with transport layer security. Technical report, Intel Corporation, 2018.
  • [18] H. Krawczyk and P. Eronen. HMAC-based Extract-and-Expand Key Derivation Function (HKDF). RFC 5869 (Informational), May 2010.
  • [19] Dayeol Lee, David Kohlbrenner, Shweta Shinde, Dawn Song, and Krste Asanović. Keystone: An open framework for architecting tees. Technical report, UC Berkeley, 2019.
  • [20] C. Liu, X. S. Wang, K. Nayak, Y. Huang, and E. Shi. Oblivm: A programming framework for secure computation. In 2015 IEEE Symposium on Security and Privacy, pages 359–376, May 2015.
  • [21] Jed Liu, Owen Arden, Michael D. George, and Andrew C. Myers. Fabric: Building open distributed systems securely by construction. J. Computer Security, 25(4–5):319–321, May 2017.
  • [22] John M. Code Sample: Intel Software Guard Extensions Remote Attestation End-to-End Example, 2018.
  • [23] Mark Russinovich, Edward Ashton, Christine Avanessians, Miguel Castro, Amaury Chamayou, Sylvan Clebsch, Manuel Costa, Cédric Fournet, Matthew Kerner, Sid Krishna, et al. Ccf: A framework for building confidential verifiable replicated services. Technical report, Technical Report MSR-TR-2019-16, Microsoft, 2019.
  • [24] J. Salowey, H. Zhou, P. Eronen, and H. Tschofenig. Transport Layer Security (TLS) Session Resumption without Server-Side State. RFC 5077 (Proposed Standard), January 2008.
  • [25] F. Schuster, M. Costa, C. Fournet, C. Gkantsidis, M. Peinado, G. Mainar-Ruiz, and M. Russinovich. Vc3: Trustworthy data analytics in the cloud using sgx. In 2015 IEEE Symposium on Security and Privacy, pages 38–54. IEEE, May 2015.
  • [26] Shweta Shinde, Dat Le Tien, Shruti Tople, and Prateek Saxena. Panoply: Low-tcb linux applications with sgx enclaves. In NDSS, 2017.
  • [27] Deian Stefan, Alejandro Russo, John C. Mitchell, and David Mazières. Flexible dynamic information flow control in Haskell. In Haskell Symposium. ACM SIGPLAN, September 2011.
  • [28] Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. In ACM SIGCOMM, pages 149–160, August 2001.
  • [29] Mingshen Sun et al. MesaTEE, 2019.
  • [30] Mingshen Sun et al. Mutual Attestation: Why and How, July 2019.
  • [31] Huibo Wang, Pei Wang, Yu Ding, Mingshen Sun, Yiming Jing, Ran Duan, Long Li, Yulong Zhang, Tao Wei, and Zhiqiang Lin. Towards memory safe enclave programming with rust-sgx. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS ’19, pages 2333–2350, New York, NY, USA, 2019. ACM.
  • [32] Samee Zahur and David Evans. Obliv-c: A language for extensible data-oblivious computation. IACR Cryptology ePrint Archive, 2015:1153, 2015.