This document describes Confidential Space's security
controls and how the system is designed to mitigate a wide range of threats. Confidential Space is designed to let parties share confidential
data (for example, regulated data or personal identifiable information (PII))
with a workload while they retain the confidentiality and ownership of the data.
Confidential Space helps create isolation so that data is only
visible to the workload and the original owners of the data.
You can use Confidential Space for scenarios when you can't
establish trust with the workload operator or between the original parties that
created the confidential data. For example, financial institutions can use
Confidential Space to collaborate with each other to identify
fraud or analyze money laundering activities. Confidential Space
permits analysis across customer data sets, while keeping customer identities
private.
Components of a Confidential Space system
Confidential Space uses a
trusted execution environment (TEE)
that is designed to release secrets only to authorized workloads. An attestation
process and hardened OS image help protect the workload and the data that the
workload processes from an untrusted operator.
A Confidential Space system has three core components:
- A workload:
a containerized image with a hardened OS that runs in a
cloud-based TEE. You use
Confidential Computing
as the TEE that offers hardware isolation and remote attestation
capabilities.
- An
attestation service
:
an OpenID Connect (OIDC) token provider. You use this service to verify the
attestation quotes for TEE and release authentication tokens. The tokens
contain identification attributes for the workload.
- A protected resource:
a managed cloud resource such as a Cloud Key Management Service
key or Cloud Storage bucket. The resource is protected by an allow policy
that grants access to authorized
federated identity tokens
.
An intermediate step, using workload identity pools, transforms the OIDC
tokens to federated identity tokens that IAM can consume.
The system helps ensure that access to protected resources is granted only to
authorized workloads. Furthermore, Confidential Space helps
protect the workload from inspection and tampering, before and after
attestation.
In a Confidential Space system, there are three parties:
- The workload author
, who creates a containerized image that includes an
application that has access to the protected resources. The author doesn't
have access to the data or the results. In addition, the author doesn't
control access to the data or the results.
- The workload operator
, who runs the workload in a Google Cloud project. The
operator typically has full administrative privileges to the project. The
operator can manage Google Cloud resources such as Compute Engine
instances, disks, and networking rules, and the operator can interact with
any Google Cloud API that acts on them. The operator doesn't have access to
the data or the results, and can't influence or modify the code or execution
environment. In addition, the operator doesn't control access to the data or
the results.
- The resource owners
(or data collaborators), who own the protected
resource. A resource owner can access their own data, set permissions on
their own data, and access the results. They can't access the data from
other resource owners or modify the code by themselves.
Confidential Space supports a trust model where the workload
author, workload operator, and resource owners are separate, mutually
distrusting parties.
The following diagram shows the system components and parties. The workload is
located in a separate project from the protected resource.
Examples of secure data processing
Confidential Space helps you to preserve a user's privacy while
sharing data. The following table describes three example use cases.
Use case
|
Example scenario
|
Functional encryption model
|
In a functional encryption model, Alice wants to share the result of
her confidential data with Bob, without revealing her entire dataset.
Alice encrypts her dataset, and protects the data encryption key
(DEK) in Cloud KMS in her project. Alice writes a program
that implements the workload, and shares the binary with Bob. Alice configures KMS to give the program access to the DEK. The
workload runs in Bob's Confidential Space, decrypts
and processes Alice's dataset, and writes the result to Bob's
Cloud Storage.
|
Multi-party computation
|
In multi-party computation, Alice and Bob want to share the result
with each other, while preserving the confidentiality of input
datasets. Similar to the functional encryption model, Alice and Bob
encrypt their respective datasets, and protect the DEKs in the
Cloud KMS instances in their projects. They co-author a
program that determines the results, and run it in a
Confidential Space. Alice and Bob configure KMS to give the program access to the DEKs. The program reads and processes
both datasets, and writes the result to a shared
Cloud Storage bucket.
|
Key shares
|
A more complex scheme uses the idea of
key shares
. A key
share is a private key that is split across Alice, Bob, and other owners such
that knowledge of individual shares doesn't give access to the
encrypted dataset. In this scheme, trust is split across multiple
owners. The private key is assembled only in a restricted TEE, by an
authorized workload.
|
In these examples, only the workload has access to the encrypted dataset and can
process it. Confidential Space helps ensure that no one can carry
out unaudited operations on data that they don't own. Data owners control how
their data is used, and which workloads are authorized to act on it.
Protecting the integrity and confidentiality of a workload
To help protect the workload from an untrusted workload operator,
Confidential Space implements the following security controls:
- An
attestation process
detects modifications to the workload image or its
TEE. This control helps protect the workload's integrity pre-attestation.
- A
hardened base image
helps reduce the attack surface, and helps prevent the
workload operator from accessing or compromising the instance at runtime.
This control helps protect a workload's integrity and confidentiality
post-attestation. Together, these security controls help protect the
workload, its secrets, and the data it processes.
Attestation process
The attestation process is based on
Shielded VM
measured boot and extended runtime measurements. This process captures boot
sequence measurements in a protected, extend-only register in the virtual
Trusted Platform Module (vTPM) device.
Measurements cover early boot components, the loaded kernel, and the container
image. Furthermore, they include environment properties such as a flag that
indicates the instance is a
Confidential VM
.
The vTPM signs (or quotes) these measurements using a certified attestation key
that is trusted by the attestation service.
The following diagram shows the components of a
Confidential Space system and how each component participates in
the attestation process.
The attestation process depends on the following components:
- Guest firmware:
an immutable component that is a trusted part of
Google Cloud.
- Attested Confidential Space image:
a hardened image based on
Container-Optimized OS
that is read from the attached boot disk.
- Early boot components:
bootloaders and kernels that interact with the
vTPM to measure the boot components into a Platform Configuration Register
(PCR).
Launcher
:
a component that downloads the workload binary from the image repository and
measures the container and its configuration into a PCR. The launcher reads
its configuration from the instance
metadata server
.
Attestation handling code:
code that is responsible for preparing a PCR
quote, and returning the vTPM's quote, attestation key, and the full
eventlog.
The
attestation service
:
a service that verifies the quote, replays the event log, issues the OIDC
token, and returns the token with the attributes for the workload access
policy.
Hardened image
The Confidential Space image is a minimal, single-purpose OS. The
image runs the container launcher, which in turn launches a single container.
The Confidential Space image builds on the
existing security
enhancements of Container-Optimized OS, and adds the following
benefits:
- Encrypted disk partitions with integrity protection:
the
Confidential Space image includes the following partitions:
- A
root-fs
partition and an OEM partition that includes the launcher
binary. These partitions are immutable and protected by
dm-verity
.
- A temporary writable partition that stores the downloaded workload
binary.
dm-crypt
encrypts this partition and protects its integrity.
- A
tmp-fs
partition that maps to RAM. In a
Confidential VM TEE, the VM's memory is encrypted. Also,
the
swap-fs
system is disabled, which helps prevent a misconfigured
operating system from storing data to disk.
- Authenticated, encrypted network connections:
the launcher uses TLS to
authenticate the attestation service, and protect its communication links.
- Various boot measurements:
these measurements include kernel
command-line arguments,
dm-verity
configuration for
root-fs
, and the
workload binary.
Disabled remote access and cloud-specific tooling:
these tools include
sshd
and
OS Login
.
Reduced state transitions:
for example, the launcher runs a single
container and then terminates.
Threat model and mitigations
This section describes the threat models that Confidential Space
helps to mitigate, and the new risks introduced by
Confidential Space.
The following attacks are outside of the
scope of this document:
- Software supply chain attacks that apply to guest Unified Extensible
Firmware Interface (UEFI) firmware, the Confidential Space
image bootloader and kernel, the container runtime, and third-party
dependencies for the workload. Data collaborators assume that resource
owners have reviewed and audited the container image before the resource
owners share their resource with an allow policy.
- Attacks on Google Cloud, such as VM escapes.
Possible attacks
Confidential Space is designed to defend against three possible
attacks:
- A malicious workload operator:
a malicious workload operator can modify
disk contents, intercept network connections, and attempt to compromise the
TEE at runtime. A malicious operator can expand the attack surface or
restrict the runtime environment. For example, a malicious operator can add
a serial console to introduce new attack vectors. As another example, a
malicious operator can constrain resources such as limiting a guest's memory
size, changing its disk space, or changing firewall rules. These constraints
might trigger I/O errors that could lead to poorly tested error cases.
- A malicious attestation client:
this attacker connects to the
attestation service and sends malformed yet signed event log messages.
- A malicious resource owner:
a malicious resource owner has full control
over the encrypted dataset that the workload consumes. This attacker might
present malformed inputs or skewed data, and attempt to trigger parsing
vulnerabilities in the workload or attempt to circumvent its privacy
controls.
Attack surfaces
The following table describes the attack surfaces that are available to
attackers.
Attacker
|
Target
|
Attack surface
|
Risks
|
Workload operator
|
TEE, Workload
|
Disk reads
|
Anything read from the disk is within the attacker's
control.
Services such as
multi-writer
persistent disks and
dynamic disk attachments mean that an attacker can modify disk
contents dynamically and at will.
|
Workload operator
|
TEE, Workload
|
Disk writes
|
Anything written to disk is visible to an attacker. See
disk snapshots
and
import
capabilities.
|
Workload operator
|
TEE, Workload
|
Metadata server
|
Instance attributes read from the metadata server are within the
attacker's control, including startup script and environment
variables.
|
Workload operator
|
TEE, Workload
|
Network
|
External network connections to the image repository or
attestation service can be intercepted. This attack is done using a
private VPC with a public-facing
Cloud Router
instance.
|
Attestation client
|
Attestation service
|
Event log and attestation messages
|
The attestation service has complex, crypto-heavy logic that is challenging to
write defensively.
|
Resource owner
|
Workload
|
Encrypted dataset
|
An attacker can poison the workload's input datasets, which means that encrypted data isn't necessarily trusted data.
|
Google Cloud infrastructure
Google Cloud includes the Compute Engine hypervisor, the vTPM for
the Confidential VM, guest UEFI firmware, and a hosted
attestation service. Sensitive key material such as vTPM and OIDC signing keys
are designed to be securely protected.
Google infrastructure is designed to logically isolate each customer's data from
the data of other customers and users, even when it's stored on the same
physical server. Administrative access for support personnel and engineers is
limited
,
audited, and
transparent
to customers. In addition, inline memory encryption in
Confidential VMs helps protect the confidentiality of the
instance memory. Inline memory encryption renders direct inspection or
accidental memory logging (kernel crash log) ineffective. For additional
information on how we protect our platform, see the
Google security overview
.
Threats and mitigations
An encrypted file system with integrity protection is designed to mitigate risks
from disk attacks. Furthermore, after code is read from disk, it's measured and
that data is never re-read from disk again. Secrets are never disclosed in
plaintext form to the disk or to any external device such as a serial console.
Risks from network attacks are mitigated by having authenticated, end-to-end
encrypted channels. External network access, such as SSH, is disabled in the
image. An attestation protocol helps protect the boot sequence and any
configuration read from the metadata server. Finally,
Confidential Space workloads are expected to use differential
privacy controls to mitigate risks from skewed datasets.
The following tables describe the threats and mitigations:
Attacks on the measured boot process
This table describes potential threats and mitigation strategies related to the
measured boot process.
Threat
|
Mitigation
|
Mitigation implementation
|
An attacker boots a Shielded VM with old firmware that
doesn't support measured boot.
If successful, an attacker may play back arbitrary
measurements and defeat remote attestation.
|
This threat is mitigated by the Google Cloud control
plane.
Confidential Space adds a vTPM device and
up-to-date UEFI firmware. Also, Confidential Space
enables measured boot, and measured boot cannot be disabled.
|
Within Google Cloud infrastructure
|
An attacker overwrites UEFI firmware in guest physical memory,
reboots the guest which resets the vTPM registers, and executes
modified firmware.
If successful, an attacker may play back arbitrary
measurements, and defeat remote attestation.
|
This threat is mitigated by the hypervisor. On guest reboot, the
hypervisor loads a clean copy of the UEFI firmware to guest memory.
Previous modifications in guest memory are discarded. Furthermore,
guest reboot is the only event that resets the vTPM.
|
Within Google Cloud and by you enabling
Confidential Computing
|
An attacker modifies unmeasured configuration files, which
negatively affects program execution.
|
This threat is mitigated by the attestation process. All
executable binaries and respective configuration files are fully
measured before running.
In particular, secure-boot variables, grub configuration, and
kernel command-line arguments are measured.
The
security
review
found that no measurements were missed in the attestation
process.
|
Within the Confidential Space image
|
An attacker triggers memory corruption vulnerabilities in early
boot components, and gains code execution.
|
Early boot components are written in the C language. These
components process untrusted user data, and might be vulnerable to
memory corruption issues. For a recent example, see
BootHole
.
This risk is mitigated by the attestation process: early boot
components must measure any user-controlled data before processing
it. A BootHole attack uses a modified
grub.cfg
to gain
code execution and defeat secure-boot.
However, in a Confidential Space system, that
attack fails to pass attestation, as
grub.cfg
measurements don't match the expected configuration.
A related risk comes from complex file system logic. Past
vulnerabilities such as
Sequoia
demonstrate that file system drivers process complex data structures,
and can be vulnerable to memory corruption issues.
This risk is
mitigated using block level
dm-verity
and
dm-crypt
integrity
protection, and by disabling auto-mounts.
|
Within the Confidential Space image
|
An attacker modifies early boot binaries on disk after they're
read and measured, before they're read and executed (disk
TOCTOU
).
|
Early boot components were built for bare metal machines, and
might not expect the cloud's dynamic environment. Boot components
might assume that disk contents cannot change during execution, which
is an incorrect assumption for cloud environments.
This risk
is mitigated by using defensive programming: disk contents are read
only once using the read, measure, execute pattern.
The
security
review
for the Confidential Space image didn't identify
TOCTOU issues in the early boot components
such as UEFI,
Shim
, or
GNU GRUB
.
|
Within the Confidential Space image
|
An attacker modifies device drivers and user mode services on
disk after the kernel is loaded.
|
This threat is mitigated by a root file system with integrity
protection.
Root-fs
in the Confidential Space image is
integrity protected by
dm-verity
. The configuration (
root-hash
) is
passed in a kernel command argument, and then measured and attested
by the attestation service.
|
Within the Confidential Space image
|
Attacks on the container launcher
This table describes potential threats and mitigation strategies related to the
launcher.
Threat
|
Mitigation
|
Mitigation implementation
|
An attacker intercepts the network connection of the launcher or
image repository.
|
The connection to the image repository is protected by an
authenticated, encrypted TLS connection.
An attacker can change the image URL and control the
workload binary. However these actions are reflected in the
attestation log.
The image repository is not controlled using an access
list, therefore the image is assumed to be viewable by everyone. You
must ensure that the workload container image doesn't contain any
secrets.
|
Within the Confidential Space image
|
An attacker modifies the workload image on disk after it was
downloaded and measured.
|
This threat is mitigated by a writable partition that is
encrypted and its integrity is protected.
The workload image and its temporary data are protected by
dm-crypt
using ephemeral, per-boot keys.
The
dm-crypt
disk encryption process does allow for rollback
attacks, where an attacker replaces disk contents with previously
seen ciphertexts and tags. These attacks are considered to be highly
complex in Confidential Space settings.
|
Within the Confidential Space image
|
An attacker modifies the launcher configuration in the metadata
server, and controls the image repository URL.
|
The attestation process detects unsafe configurations that load
non-authentic images.
|
Within the attestation service
|
Attacks on the attestation service
This table describes potential threats and mitigation strategies to the
attestation service.
Threat
|
Mitigation
|
Mitigation implementation
|
An attacker intercepts the network connection for the launcher
or attestation service and reads the secret token from the wire.
|
This threat is mitigated by having an authenticated, encrypted
TLS connection. This connection helps protect the token from passive
eavesdropping.
An attacker cannot impersonate the service because they are
missing the TLS key. Even if successful, the attacker cannot create
valid OIDC tokens.
An attacker cannot impersonate a valid client because the
client identity is guaranteed by the attestation protocol.
|
Within the network between your workload and the attestation
service.
|
An attacker exploits parsing discrepancies, which leads to
undetected changes in the attestation process.
|
This threat can occur because the measurements event log is
serialized and sent to the attestation service, where it's parsed and
processed.
A parsing discrepancy happens when the service and the runtime
OS don't agree on the semantics of the log.
For example, if the
cmdline
field has a list of
arguments separated by a comma, then a string like
a=1, b=2
c='3,d=e'
(note the
,d
in the value substring)
might be parsed differently by the kernel and the service.
This risk is mitigated by having a parsing engine that correctly
reflects the OS behavior. In particular, the
cmdline
is
processed as a whole string, and isn't broken up to key-value
pairs.
|
Within the Confidential Space image
|
An attacker uses all service resources, which brings the service
down in a denial of service (DoS) attack. The service is interrupted
for other Confidential Space users.
|
This reliability risk is mitigated by having a distributed,
elastic service that can be easily replicated and scaled out as
needed.
Code prevents resource exhaustion by malicious
clients.
|
Within the workload
|
Attacks on workloads
This table describes potential threats and mitigation strategies related to
workloads.
Threat
|
Mitigation
|
Mitigation implementation
|
An attacker reads runtime secrets from the writable
partition.
|
This threat is mitigated with an encrypted file system. The
writable file system is mounted using
dm-crypt
. Swap is disabled,
and memory contents are not paged to
disk.
As a defense-in-depth technique, OIDC tokens are scoped and
short-lived.
|
Within the Confidential Space image
|
An attacker reads runtime secrets from the
serial console
.
|
This threat is mitigated in the
Confidential Space image because credentials and
tokens are never printed to the serial console. Furthermore, cloud
logging is disabled.
|
Within the Confidential Space image
|
An attacker updates authorized SSH keys using the
OSLogin
package, and then connects
to the running instance.
|
This threat is mitigated in the
Confidential Space image because the default
systemd
services are masked, including
sshd
.
|
Within the Confidential Space image
|
An attacker updates startup scripts in the metadata server,
which are automatically loaded by the
guest
agent
.
|
This threat is mitigated in the
Confidential Space image because the guest agent
package is disabled.
|
Within the Confidential Space image
|
An attacker pushes bad packages to the VM using the
OS config
agent.
|
This threat is mitigated in the
Confidential Space image because the OS config agent
is disabled.
|
Within the Confidential Space image
|
An attacker passes a malformed and encrypted dataset to the
workload.
|
This threat is mitigated by having defensive parsing code in the
workload.
|
Within the workload
|
An attacker passes a skewed or poisoned dataset to the workload
and attempts to learn information about the datasets from other
parties.
|
This threat is mitigated by implementing differential privacy
controls in the workload.
|
Within the workload
|
Security tests
Confidential Space went through a comprehensive security review
process at Google. This security review process included the following tests and
audits:
Negative flow end-to-end integration tests
These tests verified that
attestation fails on bad measurements, such as when the code runs in an
unexpected TEE environment or launches a modified workload container.
Manual audit of measured boot process
This review focused on
identifying missing measurements and double-read bugs. Tests verified that
code was written with security best practices in mind, and, when there was a
failure, the code closed (shut down).
Manual audit of the build process for the Confidential Space image and
launcher logic:
This review focused on package removal and attack surface
reduction.
Manual audit of the attestation service
This review focused on the
implementation of the correct attestation protocol, and avoiding parsing
issues.
Cryptography review by cyber-security experts
This review focused on
the attestation protocol, file system encryption, and integrity solutions.
Privacy review by privacy experts
This review focused on the
differential privacy controls in workloads that were authored by Google.
Continuous fuzz tests
These tests covered security critical components such
as the vTPM and attestation service.
NCC Group
,
an external pentesting organization, performed penetration tests on the system.
NCC
reviewed Confidential Space
as a secure compute platform.
What's next