Trusted execution environments (TEEs) such as Intel SGX [21, 28] enable developers to create execution areas, called enclaves with enhanced protection within a CPU. At its core, a TEE aims to provide confidentiality of sensitive application code and data in the presence of untrusted or vulnerable system software. In other words, the goal of TEEs is to allow developers to reduce the TCB of applications by placing only the most security-critical functionality in the enclave, and leaving the majority of application function the untrusted context.
Yet, deploying existing applications to TEEs typically requires that developers significantly re-architect their applications. In the case of Intel SGX, for instance, applications may not make direct system calls from within an enclave. Thus, Intel SGX application development has been shifting towards a containerized model, in which unmodified applications run on top of a TEE-specific library OS (libOS) inside an enclave [11, 40, 3, 5, 41]. These libOSes provide a general system call interface that transparently handles all enclave-to-untrusted transitions needed to call system software.
However, this practice vastly increases the TCB of an application, which now includes any untrusted third-party code as well as the underlying TEE libOS. Including third-party libraries within an enclave is problematic for two main reasons. First, application programmers rarely have the time, expertise, or even authority, to prioritize security and privacy in their development process [4, 1, 2]. Thus, developers cannot be expected to fully inspect the source code of every library they need for their application, meaning that any vulnerabilities or bugs in imported third-party software may remain undetected. Indeed, software supply chain attacks are becoming increasingly common, as adversaries leverage the widespread use of open-source libraries to disseminate malicious code (e.g., [10, 14]).
Nevertheless, the convenience of third-party libraries makes them indispensable to today’s software development practices. Developers rely on community trust, especially in open-source libraries, hoping that the third party whose code they import has not published malicious code and otherwise done their due diligence to eliminate any security vulnerabilities. For example, in spite of numerous examples of data leak bugs in security-critical libraries such as OpenSSL , developers still opt to include such libraries in their TEE application.
Second, by including unvetted third-party code in their TEE application, developers are no longer using TEEs for their originally intended purpose: TEEs are designed to run a small set of trusted application components, which do not need to be run with different privileges. In a large-TCB application, in which different components may require different access privileges to sensitive data, all code running inside an enclave still runs with the same privileges. As such, all third-party code imported into an application running inside a TEE libOS container has unfettered access to all enclave memory leaving the application susceptible to data leaks and corruption.
To make matters worse, most TEE libOSes proposed thus far do not enforce privilege isolation. This crucial security mechanism in traditional OSes distinguishes between user-level and kernel-level processes, ensuring that userspace processes cannot access kernel-level memory. On the other hand, TEE libOSes execute in userspace alongside the application they are running. So, despite running inside a TEE, the libOS’s internal management data structures, such as the file descriptor table or the mount table, may be corrupted by vulnerable or malicious third-party code imported by the application developer.
To enable developers to reap the benefits of TEEs while making their applications more robust in the face of unvetted third-party libraries, we present EnclaveDom, an in-enclave privilege separation system for large-TCB applications running inside TEEs.
Privilege separation in legacy applications has been the subject of a large body of prior proposals, all with the common goal of enforcing least privilege . Similar to approaches that leverage process isolation to restrict different application components’ access to sensitive data or OS resources (e.g., [8, 43, 9]), prior research addressing privilege separation in Intel SGX applications, for instance, has proposed running individual application components in multiple separate Intel SGX processes with communicating enclaves [11, 40, 19].
Yet, these systems either only control access to sensitive data at a per-enclave granularity [11, 19], or they do not protect the TEE libOS or shim layer itself against application code [11, 40]. Furthermore, this approach requires inter-enclave communication between the different processes to compute on shared data, which incurs significant performance overheads.
In contrast, EnclaveDom subdivides a single enclave into multiple isolated memory compartments, each with its own access policy. EnclaveDom then stores developer-specified sensitive in-enclave data objects, such as TLS keys or sensitive datasets, in these compartments. Thus, EnclaveDom can enforce fine-grained least privilege at the granularity of in-enclave functions without refactoring the application into multiple TEE processes.
One major challenge in EnclaveDom’s single-enclave design is to securely share sensitive enclave data between functions with different access privileges to the same data. EnclaveDom addresses this issue using hardware-assisted memory tagging. Thus, EnclaveDom creates different memory domains within an enclave by assigning memory tags to enclave pages.
While the mechanisms we present in this paper apply generally to TEEs, we realize EnclaveDom on top of Intel SGX using Memory Protection Keys (MPK), a memory tagging technique developed at Intel. MPK provides a special hardware register that stores a process’ access privileges to each MPK tag, and enforces two types of access privileges to tagged pages (read-only and read-write). Userspace processes may then dynamically adjust the access privileges for different memory tags.
EnclaveDom targets two Intel SGX deployment settings: (1) containerized applications running in a TEE libOS, and (2) Intel SGX-native applications developed using the Intel SGX SDK . Although Intel SGX-native applications typically have a smaller TCB than the libOS setting, developers often still incorporate unvetted third-party libraries into their enclave code for ease of development. Thus, in both settings, EnclaveDom helps developers enhance the protection of sensitive in-enclave data against leaks or tampering by third-party libraries running in the enclave. In the TEE-libOS setting, EnclaveDom can additionally be used to implement OS privilege isolation.
To declare sensitive in-enclave data objects and define those in-enclave functions that are authorized to access them, developers in EnclaveDom specify the sensitive input arguments or return values of third-party library functions executing inside in the enclave in a central policy file. At run time, EnclaveDom then assigns each developer-specified data object to an MPK memory domain, and only grants access to those domains when a privileged in-enclave function is executing.
EnclaveDom does not require extensive manual annotations to application source code by the developer, and can be integrated into TEE libOSes to transparently improve the protection of the containerized applications they run. We demonstrate the effectiveness of our approach by porting EnclaveDom to the Graphene-SGX libOS . As a preliminary step, our prototype implements privilege isolation in Graphene-SGX to help protect the internal libOS data structures, including the file descriptor and mount point tables, against leaks and tampering by untrusted application-level code running on top of the libOS in the enclave.
Although we needed to modify the Graphene-SGX source code to add support for EnclaveDom, applications still run completely unmodified inside the TEE libOS container. To analyze the security properties of EnclaveDom, we first study an adversarial application that mounts a libOS data corruption attack in our Graphene-SGX prototype, and then examine EnclaveDom’s ability to help prevent additional hypothetical data leak and corruption attacks mounted within enclave code. We evaluated our EnclaveDom prototype’s performance via Graphene-SGX system call microbenchmarks, and find that EnclaveDom imposes acceptable performance and modest memory overheads.
We instantiate the EnclaveDom memory access control system as an experimental combination of Intel SGX and Memory Protection Keys. This section describes these two techniques, and summarizes the programming framework for Intel SGX applications.
2.1 Intel® Sgx
Intel Software Guard Extensions (SGX) [21, 28] is a trusted execution environment technology that is designed to preserve the confidentiality of application code and data, even in the face of untrusted or compromised system software.
At its core, Intel SGX provides an isolated memory region within the address space of a userspace process, called an enclave. All code and data within the enclave is encrypted for the entire lifetime of the application. This design divides an application into a trusted and an untrusted context, where the trusted context in the enclave is isolated even from the underlying OS or VMM.
Intel SGX dictates that the enclave must be entered via pre-defined entry points called ecalls, and requires that all system calls be made through special trampoline functions that exit the enclave, called ocalls. At every ecall and ocall, Intel SGX encrypts or decrypts any memory contents being passed between the contexts. Further, Intel SGX cryptographically computes an enclave measurement at application startup, which Intel SGX applications may use to attest to the authenticity of the enclave code. Fig. 1 shows the high-level memory layout of an Intel SGX application.
2.2 LibOS-based Containers for Intel® Sgx
Intel SGX-specific library OSes (libOS) such as Graphene-SGX , SCONE , Panoply , and SGXKernel , are designed to facilitate the deployment of Intel SGX applications by transparently handling all enclave-to-untrusted transitions necessary to utilize standard OS features. To this end, libOSes provide a trusted shim layer that implements a full userspace-level system call interface essentially creating an in-enclave container that runs unmodified applications on top of the libOS. As an example, Fig. 2 shows the system architecture for the Graphene-SGX libOS.
We present a prototype of EnclaveDom based on Graphene-SGX, but emphasize that our mechanisms are not specific to Graphene-SGX. One key feature of Graphene-SGX is its support of dynamically loaded code, requiring only very few additional steps to develop an application for Graphene-SGX. To deploy an application, developers must first specify the resources (e.g., files and network rules) required by the application in an application-specific manifest file. Following Intel SGX requirements, the manifest also specifies certain enclave parameters, such as the maximum enclave size and maximum number of threads, which developers may configure based on their application’s needs. Developers may then run a Linux executable via the Graphene-SGX command-line tool.
2.3 Intel® SGX-native Software Development
In order to provide Intel SGX’s security properties, Intel SGX requires that programmers follow two main development rules when creating an Intel SGX-native application.
First, all enclave code subject to enclave measurement must be static, i.e., built as a statically linked library as part of the whole application. In other words, a major task facing Intel SGX application developers is defining a priori all of the code that runs within the enclave, as well as the ecalls and ocalls for transitioning into and out of the enclave.
To alleviate this effort, the Intel SGX SDK  provides the Edger8r tool which automatically generates ecall and ocall trampoline functions based on a developer-specified enclave definition file specifying the function signatures for the functions implementing the enclave entry and exit interface. More specifically, Edger8r generates the source and header files for the pre-defined trampoline code, which can then be included as part of the enclave code.
Second, enclave code must be digitally signed to allow Intel SGX to verify the enclave measurement as well as verify the legitimacy of the enclave author. To this end, the Intel SGX SDK also provides the Sign Tool that automatically computes the enclave measurement and corresponding signature given the enclave code and the application developer’s enclave signing key. Upon enclave startup, Intel SGX can then verify the measurement and signature to confirm the authenticity of the application’s enclave.
2.4 Memory Protection Keys (MPK)
Memory Protection Keys (MPK)  are a recently-developed feature of the Intel x86 architecture, which enables userspace processes to tag each page table entry with a 4-bit protection key. Thus, applications create up to 16 isolated memory domains within their process address space, and assign individual pages to different domains.
MPK introduces the PKRU, a per-core 32-bit register that stores a bitmap of read-only and read-write access bits for the 16 protection keys. Whenever a process requests access to a page, the memory management unit checks the PKRU to verify that the process has sufficient privileges to access the requested address. To adjust the access permissions to a given domain, an application calls the WRPKRU instruction. Though not available on any publicly released CPUs at the time of writing, MPK may also be used to tag Intel SGX enclave pages, providing an additional layer of security to Intel SGX enclave memory.
3 System Model and Design Goals
The goal of EnclaveDom is to provide a privilege separation system that helps protect large-TCB applications running inside a TEE, i.e., containerized applications running inside a TEE libOS as well as large TEE-native applications, against data leaks and corruption by in-enclave third-party code. EnclaveDom combines the security properties of TEEs and hardware-enforced memory tagging to isolate sensitive in-enclave data objects, and only allow privileged in-enclave functions to access these data objects.
3.1 Threat Model
As in TEEs, EnclaveDom does not trust any hardware outside the CPU, system software including the OS and any hypervisor, other applications running on the same machine alongside the TEE application, as well as the untrusted component of the application running outside the enclave.
However, EnclaveDom assumes a large TCB programming model for TEEs. That is, application developers wish to deploy TEE-protected applications while incorporating third-party libraries that have not been fully vetted (if at all). Thus, while the TEE application developer may have good intentions, EnclaveDom does not trust third-party code the developer imports into her application as part of the enclave code, since it may contain undiscovered data leak vulnerabilities or attacks. Nevertheless, EnclaveDom maintains trust in the TEE libOS container running unmodified applications, as well as in the Intel SGX SDK and its tools used to implement Intel SGX-native applications.
Importantly, EnclaveDom does not aim to mitigate control flow attacks such as buffer overflow or ROP attacks. Such attacks are typically performed as a pathway for an adversary to gain control over the application and execute attack code. Yet, because the adversary’s code has already inadvertently been included in the enclave via a third-party library, EnclaveDom assumes that this is sufficient to allow the attacker to moderate the control flow of the enclave. Thus, EnclaveDom’s primary goal is to prevent unauthorized data accesses that lead to data leaks, corruption or enable other security attacks. Techniques complementary to our approach (e.g., [25, 20]) may be employed to safeguard applications against control flow attacks.
EnclaveDom also does not address side channel or enclave API misuse vulnerabilities such as Iago attacks [12, 19]; countermeasures to these vulnerabilities have been proposed in prior research (e.g., [19, 26, 39, 31, 15]) and are complementary to our work. Much like prior proposals, we also do not address denial-of-service attacks.
3.2 Security Properties
Our design for EnclaveDom provides the following three security properties.
P1: Least Privilege. An in-enclave function may only access those sensitive enclave data objects that this function needs to provide its expected functionality. EnclaveDom achieves such fine-grained least privilege by placing developer-specified sensitive enclave data objects in isolated memory compartments, and enforcing a default-deny access policy to these compartments at the granularity of individual functions running inside the enclave.
P2: Single Enclave Isolation. Strong isolation of sensitive in-enclave data does not require partitioning a TEE application into multiple enclaves. EnclaveDom relies on a hardware-assisted memory isolation technique capable of controlling access to individual enclave pages, enabling the creation of isolated memory compartments within a single enclave.
P3: Secure Data Sharing. Multiple in-enclave functions may need to operate on the same sensitive data object. To enable data sharing between functions that may have different access privileges within a single enclave, EnclaveDom’s memory compartments support dynamic access privileges. That is, EnclaveDom enforces access permissions to a given sensitive data object based on the function requesting access.
Non-goals. EnclaveDom automatically controls access to sensitive data inside a TEE at the level of in-enclave functions. However, EnclaveDom does not aim to provide automated application code partitioning as in Glamdring . Such execution isolation is orthogonal to our approach, and could be used in conjunction with EnclaveDom to further reduce the TCB of TEE applications. We assume that the application developer has partitioned her application into trusted and untrusted components either manually or using an automated technique.
EnclaveDom also does not ensure the correctness of the sensitive in-enclave data it helps protect; While providing these stronger security features may improve the integrity of sensitive enclave inputs and help prevent additional data leaks via buggy or malicious third-party code at run time, formally verifying the implementation of enclave code, as well as the provenance of enclave inputs, is orthogonal to EnclaveDom’s goals.
4 EnclaveDom Design
Our design for EnclaveDom provides isolation and access control for sensitive in-enclave data objects in the face of untrusted third-party enclave code, without partitioning the TEE application into multiple enclaves. In §5, we describe how EnclaveDom does not require extensive manual annotations to application source code by the developer, and how libOSes can integrate EnclaveDom to transparently improve the protection of the containerized applications they run.
Much as in prior research [30, 35], EnclaveDom relies on the application developer’s understanding of the libraries that they import, and on their high-level expectations of how these libraries access sensitive application data. Based on this information, EnclaveDom requires developers to specify access privileges for individual in-enclave functions to specific sensitive data objects.
At enclave startup, EnclaveDom maps these data objects to in-enclave memory domains §4.1 and partitions the enclave into corresponding compartments based on the developer’s policy §4.2. Then at run time, EnclaveDom monitors the enclave execution, and dynamically adjusts the enclave’s access to the memory domains according to the currently executing function §4.3. Figure 3 shows the main components of the EnclaveDom system.
EnclaveDom introduces three primitives that aim to provide usable yet strong protection improvement of in-enclave data.
A sensitive data object represents a unit of in-enclave data that developers wish to protect. Each data object has an associated label, a human-readable string uniquely identifying an object in the policy and EnclaveDom run-time monitor. Examples of sensitive data objects might be a cryptographic key in an encrypted data processing application such as Opaque , or a data buffer containing a genome data set in a genomics application.
EnclaveDom uses memory domains to represent isolated memory compartments that hold one or more sensitive data objects within an enclave address space. Each domain is then mapped to its own MPK tag to control access to the domain at run time. To specify in which domain EnclaveDom will place a given data object, developers may also assign a unique label to each domain in their policy.
At run time, EnclaveDom creates a dynamic execution sandbox, to execute a given in-enclave function with temporarily elevated access privileges to those memory domains specified in the developer’s policy. Upon returning from this function, EnclaveDom immediately revokes access to those memory domains to limit any further access outside the scope of the privileged in-enclave function.
4.2 Enclave Compartmentalization
The EnclaveDom Policy Manager maintains an access control list (ACL) based on the developer-supplied data access policy. As we show in Fig. 4, at enclave startup, the EnclaveDom Allocator partitions the enclave into memory domains based on the ACL, and is then responsible for allocating sensitive data objects in their corresponding domains at run time.
Since EnclaveDom aims to improve the protection of a wide variety of sensitive data objects, memory domains support flexible data object sizes. To this end, each memory domain consists of a pool of enclave pages all tagged with the same MPK tag. Notably, developers need not know the exact size of their sensitive data objects in order to isolate them in a domain since EnclaveDom allows developers to dynamically allocate domain memory at run time.
This mechanism allows EnclaveDom to support arbitrary and/or variable data object sizes. For added flexibility, developers may configure the maximum number of pages per pool based on an estimate of the maximum memory consumption of their specified data objects.
It is important to note that there is a trade-off between the granularity of data object isolation (i.e., security) and an application’s memory footprint. Recall from §2.4 that changes to the PKRU affect all memory pages with the same MPK tag. This means that, on the one hand, if an application’s memory domains contain only single, small data objects, most of the reserved enclave pages in the domains’ pools will remain unused throughout the lifecycle of the application inflating the memory footprint of the application.
However, this policy requires that functions have sufficient privileges to access the memory domains corresponding to each data object. On the other hand, a developer may use the reserved page pool more efficiently by placing multiple data objects in the same memory domain, at the risk of potentially allowing a function that should only access one of those data object to leak another sensitive data object in the same domain.
4.3 Dynamic Sandboxed Execution
MPK restricts access to memory pages at the CPU core level meaning that all code running on the same core has access to all tagged pages whose PKRU access bits make those pages accessible. However, to enforce least privilege, EnclaveDom cannot allow enclave functions with insufficient privileges to access specific data objects.
To enhance the protection of sensitive data objects in face of leaks or tampering by an untrusted third-party library at run time, EnclaveDom helps to ensure that only those memory domains to which the currently executing in-enclave function has privileges are accessible, and denies access to all other domains. Specifically, upon entering a privileged in-enclave function, the EnclaveDom Monitor queries the Policy Manager for the given function’s data object access rules, and encapsulates the function in an execution sandbox by dynamically adjusting the access bits for the MPK tags associated with the corresponding memory domains (see Fig. 5).
After returning from the sandboxed function, and before exiting the execution sandbox, the EnclaveDom Monitor then once again restricts access to all memory domains. While this step may be avoided if the next enclave function to be executed also has access to some or all of the accessible domains, EnclaveDom takes a conservative approach and makes no assumptions about the control flow of applications.
|Data object management*|
|int enclavedom_check_input_size(const char *obj_label, size_t size)|
|int enclavedom_check_output_size(const char *obj_label, size_t size)|
|int enclavedom_copy_from_untrusted(const char *obj_label, void *untrusted_buf)|
|int enclavedom_copy_to_untrusted(const char *obj_label, void *untrusted_buf)|
|Domain memory management*|
|void *enclavedom_malloc(const char *domain_label, size_t size)|
|void enclavedom_free(const char *domain_label, void *addr)|
|int enclavedom_grant_data_access(const char *func_name)|
|void enclavedom_revoke_data_access(const char *func_name)|
|Access semantics||Object specification|
|no writable objects|
|all writable objects|
|blanket access to domain||#<domain label>:|
|skip object size verification||<object label>#<domain label>:|
We implement EnclaveDom as a userspace library that can be integrated into Intel SGX-specific libOSes and Intel SGX-native applications in a handful of steps, and we demonstrate EnclaveDom’s practicality and benefits with our EnclaveDom-protected Graphene-SGX libOS prototype.
5.1 EnclaveDom API
To facilitate adoption and usability, our userspace library for EnclaveDom exposes a small programming interface which abstracts away the use of MPK. Importantly, the execution sandboxing API calls can be automatically generated in both of EnclaveDom’s targeted deployment settings. For developers writing Intel SGX-native applications, we envision providing a modified version of the Edger8r tool to minimize the number of manual changes to application source code. Further, to allow developers of containerized applications (i.e., those running inside a libOS) to improve the protection of sensitive data via EnclaveDom, we could provide a tool for libOS developers to transparently enforce the application developer’s data access policy and automate execution sandboxing.
We also provide a policy generation tool that automatically populates header files with the internal representation of the application’s data object ACL (see §5.4). To maintain the integrity of the developer’s data access policy, we do not provide any policy-related API calls; the internal representation of the policy is only accessible within the EnclaveDom code. Table 1 shows EnclaveDom’s API.
Initialization and teardown. In order to enable EnclaveDom’s run-time memory access control, developers must use the initialization and teardown API. Notably, these functions should only be called once each, and may be called either from the untrusted application context or from within the enclave. Given the data object ACL generated at enclave build time, enclavedom_init() is responsible for pre-allocating the enclave pages reserved for memory domains, and for provisioning and mapping MPK tags to memory domains. Correspondingly, enclavedom_teardown() releases provisioned MPK tags with the OS, and frees the memory domain enclave pages. These two API calls are the only EnclaveDom functionality that libOS developers and Intel SGX-native application developers must manually incorporate into their application source code.
Data object management. Intel SGX-native application developers may wish to pass data into the enclave and treat it as sensitive data, or return a data object to the untrusted context as the result of an enclave computation. To support passing sensitive data objects during ecalls and ocalls, EnclaveDom provides developers with an API for copying a given sensitive data object into or out of the enclave. As part of passing sensitive data objects between contexts, EnclaveDom also allows developers to verify the size of the data objects. This API may only be called from within an enclave. To ease adoption, these API calls can be easily integrated into the Intel SGX SDK’s Edger8r edge routine generation tool.
Domain memory management. As we describe in §4.2, applications in EnclaveDom may allocate data in memory domains dynamically at run time. EnclaveDom’s memory management API allows libOS and Intel SGX-native application developers to either replace or complement existing malloc calls inside the enclave with domain-specific memory allocations. As with the data object management API, this functionality can be integrated into the Edger8r tool to facilitate the adoption process for Intel SGX-native applications.
Execution sandboxing. While developers must explicitly specify each sensitive data object that a privileged in-enclave function may access in their policy, EnclaveDom transparently maintains all data object-to-domain mappings. Developers then use EnclaveDom’s execution sandboxing API to specify the boundaries of a function sandbox by encapsulating privileged function calls between a enclavedom_grant_data_access() call at the entry of a privileged function, and the corresponding enclavedom_revoke_data_access() call after returning. In §5.4, we describe how EnclaveDom can help to automate this process for application developers in both deployment settings.
5.2 Policy Specification
EnclaveDom requires application developers to declare sensitive data objects and specify corresponding enclave function-level access rules in a static policy file. In addition, libOS developers may also generate a EnclaveDom policy to improve the protection of internal libOS data structures that is included in the libOS core (see §5.3). Every access rule takes the form:
where is the name of the in-enclave function affected by the given rule. specifies those sensitive data objects to which has read-only access (i.e., function inputs), and specifies those data objects to which has read-write access (e.g., function return values).
In an access rule, and are written as comma-separated lists of object specifications formatted as
<object label>#<domain label>:<object size>.
To allow developers to tailor their policies to their application’s security and performance requirements, EnclaveDom supports flexible data access semantics, detailed in Table 2.
Additionally, EnclaveDom does not require a 1:1 correspondence between data objects and memory domains. That is, multiple individual data objects may be placed into the same domain. Such access rules may be beneficial in cases in which a number of different in-enclave functions require access to the same individual data objects.
5.3 Privilege Isolation for Graphene-SGX
As described in §2.2, libOSes such as Graphene-SGX  provide application developers with a mechanism for reaping the benefits of Intel SGX, while running unmodified applications. However, Graphene-SGX’s trusted shim layer performing privileged operations (i.e., system calls) runs with the same privileges as the application binary and any third-party libraries imported into the application. This lack of privilege isolation poses a threat to the libOS and any applications it runs because vulnerabilities or attacks in untrusted third-party code may corrupt or leak sensitive internal libOS data.
As a preliminary step to demonstrate EnclaveDom’s ability to improve the protection of large-TCB applications against third-party code, we have built a EnclaveDom-enabled prototype of the Graphene-SGX libOS implementing a privilege isolation mechanism. Our current prototype enforces privilege isolation by placing all file system-related data structures into memory domains, and by only granting access to this data within relevant system calls. More specifically, we declare two memory domains: handle_dom contains all file descriptor-specific metadata, and fs_dom contains all file system management metadata (e.g., the mount table).
Our decision to create these two domains was largely based on Graphene-SGX’s implementation of the file system interface, which treats these two types of metadata separately employing a separate memory manager for each. As such, we were able to replace Graphene-SGX’s internal memory allocation calls in these memory managers with EnclaveDom domain memory management API calls for the corresponding memory domains.
We currently declare data access rules for 32 libOS system calls, for which we automatically generate the EnclaveDom policy code with our policy generator. However, porting EnclaveDom to Graphene-SGX required us to manually implement the sandboxing wrapper functions for the 32 system calls since the libOS employs a custom interface to Intel SGX; this interface is unfortunately not compatible with the official Intel SGX SDK, hindering us from automatically generating the EnclaveDom system call wrappers with the Edger8r tool. Furthermore, to comply with Graphene-SGX’s system ABI and make EnclaveDom API calls from the Library OS layer, we needed to create additional wrapper functions for EnclaveDom’s API in the Host layer.
Nevertheless, our instrumentation only affects the Graphene-SGX libOS itself, so application developers who do not wish to provide additional application-level data protections can still remain agnostic to EnclaveDom (as well as Intel SGX) by running unmodified applications on top of our EnclaveDom-enabled Graphene-SGX prototype.
5.4 EnclaveDom Code Generation
Policy generator. EnclaveDom improves the protection of the developer’s policy against tampering by untrusted code by leveraging Intel SGX’s measurement of static enclave code. To avoid having developers manually create an internal representation of their policy, we provide developers with a policy parser that automatically populates EnclaveDom’s policy data structures based on the developer’s policy, and generates a header file containing the internal representation of the data object ACL.
Developers run the policy generator as an additional step in their application build process, and include the generated ACL header file in the libOS or application source code. The EnclaveDom API can then access these policy data structures directly at run time.
Finally, the policy generator tool allows libOS and Intel SGX-native application developers to configure the number of enclave pages reserved for a memory domain. If no domain pool size is specified, the default of four 4-KB pages is used.
EnclaveDom Edger8r tool. Since most Intel SGX-native applications do not compute on static sensitive data, developers must often pass sensitive data objects between the untrusted context and the enclave at run time. To examine how much we can reduce the burden on developers and to help to ensure that EnclaveDom-protected Intel SGX-native applications enhance the protection of sensitive data objects, we built an experimental EnclaveDom-aware Edger8r tool that automatically inserts data object and domain memory allocation calls in the generated trampoline code based on the developer’s EnclaveDom policy.
In addition, EnclaveDom can bootstrap the vanilla Edger8r’s functionality to create trampoline code to create application-specific wrappers around privileged in-enclave functions to create the function execution sandboxes. These generated sandboxing wrappers include calls to EnclaveDom’s execution sandboxing API surrounding the actual call to the privileged function. Developers would then only be required to replace the calls to the original function with calls to the sandboxing wrapper function to ensure that the relevant sensitive data objects may be accessed within the scope of the sandboxed function. Any “un-sandboxed” calls to the original function would proceed but attempts to access sensitive data objects will be blocked by EnclaveDom’s Monitor.
EnclaveDom for containerized applications. While not currently implemented in our Graphene-SGX prototype, libOS developers may wish to provide support for EnclaveDom to application developers. To keep the burden on the application developers minimal, they would only be required to specify a EnclaveDom policy specifying the privileged third-party functions their application calls and the sensitive data objects these functions may access. After automatically generating the ACL header file and including this header file into their application’s source code. We then envision Graphene-SGX transparently parsing the application’s data object ACL header file, allocating the developer-specified data objects in their respective memory domains, and creating the application-specific EnclaveDom’s dynamic execution sandboxes at run time.
We evaluate EnclaveDom’s security properties by performing a case study of an adversarial application that we run in our prototype, and present an analysis of additional hypothetical vulnerabilities in enclave code. To evaluate the performance and memory overheads EnclaveDom imposes on the Graphene-SGX libOS, we ran microbenchmarks to understand how the underlying EnclaveDom operations affect our measurements.
Our tests were performed on a machine with an experimental Intel CPU with two 224 GB Intel SSDs, running Ubuntu 17.04 on Linux Kernel 4.10.0-42-generic. Intel CPUs that feature both Intel SGX and MPK are unavailable commercially at the time of writing. Further, the policy for our prototype contains access rules for 32 libOS system calls, and is configured to provision a maximum of four 4-KB memory pages per domain.
6.1 Security Analysis
To evaluate EnclaveDom’s security properties, we first do a case study on an application that attempts to tamper with Graphene-SGX’s file descriptor table. We developed this adversarial application in our laboratory setting, demonstrating the feasibility of such attacks without EnclaveDom’s protections, and EnclaveDom’s broader applicability to similar attacks. Furthermore, we also analyze additional hypothetical vulnerabilities and memory attacks in enclave code.
Case study: Descriptor Table Corruption. To demonstrate EnclaveDom’s ability to mitigate unauthorized accesses, we examine the instance of file descriptor table corruption in a libOS-based Intel SGX application. If successful, such attacks can be especially detrimental since they may enable a wide range of malicious application behaviors.
In this case study, the adversarial application imports our specially crafted malicious library which attempts to corrupt Graphene-SGX’s file descriptor table (or FD table). We ran this adversarial application in vanilla Graphene-SGX and our EnclaveDom-enabled Graphene-SGX prototype, and found that the attack succeeds in vanilla Graphene-SGX, where EnclaveDom helps prevent this attack via a segfault.
This attack is possible in vanilla Graphene-SGX because the libOS itself and the adversarial application run in the same address space. Recall from §5.3 that our prototype places the FD table in the handle_dom memory domain, which is only accessible during those libOS system calls specified in our prototype’s data access policy. In other words, since our crafted library does not make system calls that EnclaveDom executes in a sandbox, but rather attempts to access the FD table via a reverse-engineered code path that does not run inside a dynamic sandbox, access to the handle_dom is never granted to the crafted library.
While the attack in this case study may seem contrived at first glance, we believe that file descriptor table corruption demonstrates a broader vulnerability class that allows third-party code to abuse its access to in-enclave data. EnclaveDom mitigates this threat to TEE applications which arises specifically because developers include untrusted third-party code.
In addition, this case study shows that EnclaveDom can help prevent a more insidious class of attacks capable of circumventing more traditional access control methods such as system call interposition: since our studied attack does not require direct system calls to access a sensitive data object, interposition mechanisms that verify that the application is only accessing authorized OS resources would not even be invoked.
Hypothetical vulnerabilities. We consider a broad range of vulnerabilities that untrusted third-party code running inside an enclave may hypothetically be able to exploit in order to leak sensitive in-enclave data to the untrusted application component. Table 3 summarizes three hypothetical vulnerabilities, and the specific EnclaveDom mechanism that mitigates the vulnerability.
|confused deputy attack||execution sandbox||Y|
|WRPKRU misuse||static analysis||N|
|MPK OS-level API misuse||integration with libmpk ||N|
EnclaveDom allows developers to grant specific in-enclave functions privileges to access certain sensitive data objects. However, an untrusted third-party function running inside the enclave may attempt to escalate its own privileges by calling a more privileged function in order to gain unauthorized access to sensitive data objects. EnclaveDom helps prevent such confused deputy attacks  via its dynamic execution sandbox mechanism (see §4.3).
Untrusted in-enclave code may attempt to manipulate the value of the PKRU by including direct WRPKRU instructions within its code. Since MPK does not currently verify the origin of WRPKRU instructions in a userspace process, one viable mitigation strategy that EnclaveDom could employ is static analysis and binary instrumentation in order to detect and replace errant WRPKRU instructions in non-EnclaveDom enclave code (similarly as in ERIM ).
Finally, since MPK relies on the OS for provisioning protection keys, in-enclave code may attempt to gain authorized access to MPK-protected memory or corrupt the PKRU by directly making pkey_mprotect or pkey_set system calls via an ocall. Concurrent research on libmpk  proposes techniques for mitigating these issues, and may be integrated with EnclaveDom as well.
6.2 Performance Microbenchmarks
|% exec time in EnclaveDom||accessed memdom(s)|
|memory usage (in bytes)|
Execution time overhead. To analyze the impact of EnclaveDom on the performance of the Graphene-SGX libOS, we took microbenchmarks of the open, stat, fstat, mmap, and close system calls. Recall from §5 that our EnclaveDom-enabled prototype of Graphene-SGX isolates the libOS internal file descriptor and file system management data structures (e.g. mount table) into two memory domains. Thus, we chose to benchmark these five system calls since they require access either only to the file descriptor domain handle_dom, the file system management domain fs_dom, or both, and sought to determine how accessing the different memory domains affects performance.
We used the lat_syscall benchmark of LMbench 2.5 , which stress tests six system calls; for each LMbench experiment, we then measured the amount of execution time of each our target system calls spent performing EnclaveDom operations. Table 4 shows the mean percentage of execution time that EnclaveDom operations comprise in our five benchmarked syscalls as well as the EnclaveDom memory domain that each syscall accesses internally.
We observe two groups of system calls: the first group, mmap and open, spends only a small portion of execution time (at most 6.4%) performing EnclaveDom operations, while the second group of syscalls spends about 50% of the execution time performing EnclaveDom operations. This result is perhaps not entirely unsurprising given that the primary purpose of the system calls in the second group is to access the file system metadata, while open and mmap perform a much larger number of tasks beyond manipulating the internal file system data structures.
Memory overhead. Our evaluation additionally sought to quantify the memory overhead that EnclaveDom imposes on Graphene-SGX. Specifically, we measure the additional memory required to maintain the application ACL and internal memory domain management data structures for each domain as well as for Graphene-SGX. Table 5 shows the median peak memory consumption in bytes of EnclaveDom’s internal data structures for our Graphene-SGX prototype as a whole as well as each memory domain during our syscall microbenchmarks above.
We find that EnclaveDom’s memory overhead is very modest requiring only an additional 1.2 KB of memory for the entire Graphene-SGX libOS, and a mean of 0.6 KB per memory domain. Note that our configuration provisions far more memory per domain than is required.
7 Related Work
In-Intel SGX memory protection. A small number of prior proposals have sought to enhance memory protection within an Intel SGX enclave. SGXBounds  uses tagged pointers for efficient bounds-checking within an Intel SGX enclave. SGX-Shield  provides a new ASLR scheme to improve the protection of Intel SGX applications against memory corruption attacks inside the enclave. T-SGX  leverages the hardware transactional memory provided by Intel® TSX in order to help prevent controlled-channel attacks in an Intel SGX enclave. Multi-domain SFI  subdivides an Intel SGX enclave into multiple memory domains using MPX’s hardware-enforced memory bounds checking to implement privilege isolation and multi-process library OSes within a single enclave.
All of these prior proposals (except Multi-domain SFI) aim to prevent a particular class of memory attacks. In contrast, EnclaveDom operates at the granularity of application-level data objects to help developers prevent sensitive data leaks in a more intuitive fashion. Multi-domain SFI is closest to our approach, but we believe EnclaveDom’s use of MPK, a technique designed specifically for memory access control, provides a more adequate solution for our goals.
Intel hardware memory protection in legacy applications. Numerous prior proposals have leveraged novel techniques available in Intel processors to help protect sensitive application data within a single address space.
ERIM , MemSentry-PKU  and Janus  use MPK to partition a legacy application’s address space into two or more domains to isolate sensitive application data. EnclaveDom builds on this prior research generalizing the use of MPK to implement multiple memory domains and applies this mechanism to Intel SGX applications. leveraging MPK to partition an Intel SGX enclave and isolate sensitive in-enclave data objects in separate domains.
Systems such as Dune , MemSentry , and Janus  rely on Intel® VT-x virtualization hardware to create intra-process isolated compartments. Unlike EnclaveDom, the main goal of these approaches is to provide execution isolation for running different application components in separate address spaces. EnclaveDom’s main goal is to help protect sensitive application data shared between different application components by controlling access to sensitive data objects at run time.
Non-Intel hardware memory isolation. Other architectures, most notably ARM, also provide techniques designed for finer-grained memory access control. For instance, ARMlock , Shreds , and FlexDroid  leverage ARM memory domains, a technique very similar to MPK, to allow developers to create a number of isolated execution regions with associated private memory compartments.
Process isolation. A large number of prior proposals have leveraged OS primitives to restrict application components in legacy and mobile applications. Systems such as Wedge , Privtrans , Passe , CodeJail , AdSplit  and BreakApp  partition an application into multiple processes to run unprivileged components in isolated address spaces. Ryoan  partitions an Intel SGX application into separate Intel SGX processes as a means to isolate different application components.
While such process-based isolation provides strong memory protections, these approaches require significant development efforts to re-architect a monolithic application for a multi-process model. EnclaveDom’s single-process design, on the other hand, provides a more practical approach for Intel SGX application developers that does not require major application refactoring, and still provides strong hardware-assisted memory isolation.
Intra-process isolation. Arbiter , SMV , Light-weight Contexts , and Pyronia  partition a single process address space using multiple page tables in order to control access to different memory compartments at intra-process granularities (e.g., thread-level in Arbiter or SMV). EnclaveDom borrows many concepts from these intra-process isolation proposals, but relies on more efficient hardware-based techniques that are more adequate for use in Intel SGX applications.
We have presented EnclaveDom, a privilege separation system for large-TCB applications running in trusted execution environments. EnclaveDom combines the security of TEEs with memory tagging to help prevent sensitive data leaks and corruption by untrusted third-party code imported into containerized and TEE-native applications. To avoid partitioning an enclave into multiple TEE processes, EnclaveDom isolates sensitive data objects within a single enclave address space. As such, EnclaveDom supports data sharing between different in-enclave functions, while enforcing least privilege data access policies.
We have implemented EnclaveDom as a userspace API for large-TCB Intel SGX applications that uses MPK for enclave memory tagging. To demonstrate EnclaveDom’s properties, we implement OS privilege isolation in the Graphene-SGX library OS. Our evaluation of our EnclaveDom-enabled Graphene-SGX prototype imposes a very modest memory overhead and can help protect containerized applications against unauthorized accesses to sensitive in-enclave data.
We thank Mingwei Zhang, Michael Steiner, Bruno Vavala, Prakash Narayana Moorthy, Dmitrii Kuvaiskii, Mona Vij, Michael LeMay, and Vinnie Scarlata for their feedback and insightful discussions.
-  Acar, Y., Backes, M., Fahl, S., Kim, D., Mazurek, M. L., and Stransky, C. You Get Where You’re Looking for: The Impact of Information Sources on Code Security. In Proc. IEEE Symposium on Security and Privacy (May 2016).
-  Acar, Y., Fahl, S., and L. Mazurek, M. You are Not Your Developer, Either: A Research Agenda for Usable Security and Privacy Research Beyond End Users. In Proc. IEEE Cybersecurity Development (11 2016), pp. 3–8.
-  Arnautov, S., Trach, B., Gregor, F., Knauth, T., Martin, A., Priebe, C., Lind, J., Muthukumaran, D., O’Keeffe, D., Stillwell, M. L., Goltzsche, D., Eyers, D., Kapitza, R., Pietzuch, P., and Fetzer, C. SCONE: Secure Linux Containers with Intel SGX. In OSDI (2016).
-  Balebako, R., and Cranor, L. Improving App Privacy: Nudging App Developers to Protect User Privacy. In Proc. IEEE Symposium on Security and Privacy (July 2014).
-  Baumann, A., Peinado, M., and Hunt, G. Shielding Applications from an Untrusted Cloud with Haven. In OSDI (2014).
-  Belay, A., Bittau, A., Mashtizadeh, A., Terei, D., Mazières, D., and Kozyrakis, C. Dune: Safe User-level Access to Privileged CPU Features. In Proc. OSDI (2012).
-  Bittau, A., Marchenko, P., Handley, M., and Karp, B. Wedge: Splitting Applications into Reduced-Privilege Compartments. In NSDI (2008).
-  Blankstein, A., and Freedman, M. J. Automating Isolation and Least Privilege in Web Services. In IEEE Symposium on Security and Privacy (2014).
-  Brumley, D., and Song, D. Privtrans: Automatically partitioning programs for privilege separation. In USENIX Security Symposium (2004).
-  Catalin Cimpanu. Twelve malicious Python libraries found and removed from PyPI. https://www.zdnet.com/article/twelve-malicious-python-libraries-found-and-removed-from-pypi/.
-  che Tsai, C., Porter, D. E., and Vij, M. Graphene-SGX: A Practical Library OS for Unmodified Applications on SGX. In USENIX ATC (2017).
-  Checkoway, S., and Shacham, H. Iago attacks: Why the system call api is a bad untrusted rpc interface. In Proc. ASPLOS (2013).
-  Chen, Y., Reymondjohnson, S., Sun, Z., and Lu, L. Shreds: Fine-Grained Execution Units with Private Memory. In IEEE Symposium on Security and Privacy (2016).
-  Danny Grander. Malicious code found in npm package event-stream downloaded 8 million times in the past 2.5 months. https://snyk.io/blog/malicious-code-found-in-npm-package-event-stream/.
-  Gruss, D., Lettner, J., Schuster, F., Ohrimenko, O., Haller, I., and Costa, M. Strong and efficient cache side-channel protection using hardware transactional memory. In Proc. USENIX Security Symposium (2017).
-  Hardy, N. The confused deputy: (or why capabilities might have been invented). ACM Operating Systems Review 22, 4 (1988).
-  Hedayati, M., Gravani, S., Johnson, E., Criswell, J., Scott, M. L., Shen, K., and Marty, M. Janus: Intra-Process Isolation for High-Throughput Data Plane Libraries. Technical Report UR CSD/1004.
-  Hsu, T. C.-H., Hoffman, K., Eugster, P., and Payer, M. Enforcing least privilege memory views for multithreaded applications. In CCS (2016).
-  Hunt, T., Zhu, Z., Xu, Y., Peter, S., and Witchel, E. Ryoan: A distributed sandbox for untrusted computation on secret data. In OSDI (2016).
-  Intel Corporation. Control-flow Enforcement Technology Specification. https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf.
-  Intel Corporation. Intel® Software Guard Extensions (Intel® SGX). https://software.intel.com/en-us/sgx. Accessed 28 Mar 2019.
-  Intel Corporation. Intel® Software Guard Extensions (Intel® SGX) SDK. https://software.intel.com/sgx-sdk. Accessed 28 Mar 2019.
-  Intel Corporation. Intel 64 and IA-32 Architectures Software Developer’s Manual. https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf, 2018. Accessed 29 Nov 2018.
-  Koning, K., Chen, X., Bos, H., Giuffrida, C., and Athanasopoulos, E. No Need to Hide: Protecting Safe Regions on Commodity Hardware. In Proc. EuroSys (2017).
-  Kuvaiskii, D., Oleksenko, O., Arnautov, S., Trach, B., Bhatotia, P., Felber, P., and Fetzer, C. SGXBOUNDS: Memory Safety for Shielded Execution. In Proc. European Conference on Computer Systems (2017).
-  Lind, J., Priebe, C., Muthukumaran, D., O’Keeffe, D., Aublin, P.-L., Kelbert, F., Reiher, T., Goltzsche, D., Eyers, D., Kapitza, R., Fetzer, C., and Pietzuch, P. Glamdring: Automatic Application Partitioning for Intel SGX. In USENIX ATC (2017).
-  Litton, J., Vahldiek-Oberwagner, A., Elnikety, E., Garg, D., Bhattacharjee, B., and Druschel, P. Light-Weight Contexts: An OS Abstraction for Safety and Performance. In OSDI (2016).
-  McKeen, F., Alexandrovich, I., Berenzon, A., Rozas, C. V., Shafi, H., Shanbhogue, V., and Savagaonkar, U. R. Innovative Instructions and Software Model for Isolated Execution. In Proc. Hardware and Architectural Support for Security and Privacy (2013).
-  McVoy, L., and Staelin, C. Lmbench: Portable Tools for Performance Analysis. In Proc. USENIX Annual Technical Conference (1996).
-  Melara, M. S., Liu, D. H., and Freedman, M. J. Pyronia: Redesigning Least Privilege and Isolation for the Age of IoT. http://arxiv.org/1903.01950, 2019.
-  Oleksenko, O., Trach, B., Krahn, R., Silberstein, M., and Fetzer, C. Varys: Protecting SGX enclaves from practical side-channel attacks. In USENIX Annual Technical Conference (2018).
-  OpenSSL Software Foundation. OpenSSL vulnerabilities. https://www.openssl.org/news/vulnerabilities.html. Accessed 19 Nov 2018.
-  Park, S., Lee, S., Xu, W., Moon, H., and Kim, T. libmpk: Software Abstraction for Intel Memory Protection Keys. http://arxiv.org/1811.07276, 2018.
-  Saltzer, J. H., and Schroeder, M. D. The protection of information in computer systems. Proceedings of the IEEE 63, 9 (1975).
-  Seo, J., Kim, D., Cho, D., Kim, T., Shin, I., and Jiang, X. FlexDroid: Enforcing In-App Privilege Separation in Android. In NDSS (2016).
-  Seo, J., Lee, B., Kim, S., Shih, M.-W., Shin, I., Han, D., and Kim, T. Sgx-shield: Enabling address space layout randomization for sgx programs. In NDSS (2017).
-  Shekhar, S., Dietz, M., and Wallach, D. S. AdSplit: Separating smartphone advertising from applications. In USENIX Security Symposium (2012).
-  Shen, Y., Chen, Y., Chen, K., Tian, H., and Yan, S. To isolate, or to share?: That is a question for intel sgx. In APSys (2018).
-  Shih, M.-W., Lee, S., Kim, T., and Peinado, M. T-sgx: Eradicating controlled-channel attacks against enclave programs. In Proc. Network and Distributed System Security Symposium (2017).
-  Shinde, S., Tien, D. L., Tople, S., and Saxena, P. Panoply: Low-tcb linux applications with SGX enclaves. In Proc. Network and Distributed System Security Symposium (2017).
-  Tian, H., Zhang, Y., Xing, C., and Yan, S. SGXKernel: A Library Operating System Optimized for Intel SGX. In Proc. Computing Frontiers Conference (2017).
-  Vahldiek-Oberwagner, A., Elnikety, E., Garg, D., and Druschel, P. ERIM: Secure and Efficient In-process Isolation with Memory Protection Keys. http://arxiv.org/abs/1801.06822, 2018.
-  Vasilakis, N., Karel, B., Roessler, N., Dautenhahn, N., DeHon, A., and Smith, J. M. BreakApp: Automated, flexible application compartmentalization. In Proc. Network and Distributed System Security Symposium (2018).
-  Wang, J., Xiong, X., and Liu, P. Between Mutual Trust and Mutual Distrust: Practical Fine-grained Privilege Separation in Multithreaded Applications. In USENIX ATC) (2015).
-  Wu, Y., Sathyanarayan, S., Yap, R. H., and Liang, Z. Codejail: Application-transparent Isolation of Libraries with Tight Program Interactions. In ESORICS (2012).
-  Zheng, W., Dave, A., Beekman, J. G., Popa, R. A., Gonzalez, J. E., and Stoica, I. Opaque: An Oblivious and Encrypted Distributed Analytics Platform. In USENIX NSDI (2017).
-  Zhou, Y., Wang, X., Chen, Y., and Wang, Z. ARMlock: Hardware-based Fault Isolation for ARM. In Proc. ACM Conference on Computer and Communications Security (2014).