How to Deploy Confidential Computing (TEEs) for Sensitive AI Model Inference
A step-by-step deployment procedure for running LLM inference inside a hardware-backed TEE, not a vendor comparison
Proactive Security for the AI Era
NodeZero continuously and autonomously pentests infrastructure, identity, cloud, and now web applications, chaining weaknesses across every domain the way real attackers do. Every finding ships with replayable proof showing exploitable business impact, not theoretical risk.
A team has already decided it needs confidential computing: model weights are proprietary, inference inputs are regulated data, or a customer contract requires that the cloud provider's own operators be excluded from the trust boundary. The vendor decision is made. What is missing is the procedure, because standing up an inference server inside a trusted execution environment (TEE) is not the same operation as launching a normal VM and installing a model server on it. The workload has to run inside a confidential VM or a CC-enabled GPU, the deployment has to produce a cryptographic attestation report before it is trusted with any real data, and a relying party (a key management service, an orchestrator, or the client sending inference requests) has to verify that report against the hardware vendor's or cloud's attestation service before releasing secrets or sending sensitive prompts. This guide walks through that procedure end to end: prerequisites, the numbered deployment steps for an AMD SEV-SNP or Intel TDX confidential VM with an NVIDIA H100/H200 GPU in confidential computing mode on Azure, how attestation is actually wired up, how to validate that the deployment is actually protecting anything, and the failure modes and tradeoffs the deployment does not remove. For the underlying decision of which cloud and hardware combination to build on, see our companion confidential computing platform comparison of AWS Nitro Enclaves, Azure Confidential VMs, and NVIDIA Confidential Computing; this guide assumes that decision is already made and shows the practitioner how to actually deploy on it. For hardening the inference server itself once it is deployed, whether inside a TEE or not, see our vLLM production security hardening guide; the two guides are complementary rather than overlapping, since the TEE boundary and the inference server's own attack surface are separate problems.
The Problem: Encrypted at Rest and in Transit Is Not Encrypted in Use
Standard cloud security controls encrypt a model's weights on disk and encrypt inference traffic on the wire, but the moment weights are loaded into GPU memory and a prompt is decrypted for processing, both sit in plaintext in system and GPU memory for the life of the request. On a normal VM or GPU instance, that plaintext is visible to anyone with access to the hypervisor, the host OS, or physical DMA access to the machine, which in a public cloud means the cloud provider's own privileged operators and, in a worst case, a compromised hypervisor or a malicious co-tenant escaping isolation. For a team fine-tuning a proprietary model on customer data, serving inference over regulated health or financial records, or operating under a contract that specifically names the cloud operator as outside the trust boundary, that gap is the one confidential computing is built to close. A hardware-backed TEE encrypts memory with a key that never leaves the processor package (and, for GPU workloads, the GPU package), so even someone with root on the host or physical access to the box cannot read the plaintext contents of a properly attested confidential VM or GPU. It does not replace network security, access control, or supply chain integrity for the model artifact itself; it adds one specific guarantee (memory confidentiality and integrity against a privileged host or physical attacker) that nothing else in a standard cloud deployment provides.
Prerequisites
Before starting the deployment, confirm each of the following. Skipping any of these is the most common reason teams get partway through a confidential computing rollout and then have to restart on different hardware.
Hardware generation that actually supports CC mode
For GPU inference, confidential computing is supported on NVIDIA H100 SXM5 and PCIe (Hopper) and H200 SXM5, which inherits H100's Hopper-based CC support; Blackwell-generation B200 and GB200 parts extend it further. Older A100 or consumer GPUs do not support GPU confidential computing at all, so verify the specific SKU before provisioning, not just the GPU family name.
A CPU-side TEE underneath the GPU
NVIDIA's confidential computing mode does not stand alone; it requires a host CPU TEE (AMD SEV-SNP on EPYC Genoa/Milan-class hardware, or Intel TDX, which succeeded SGX as Intel's VM-level confidential computing technology for this use case) to establish the confidential VM the GPU is attached to. Confirm your cloud region and instance family actually pair the GPU with a CC-capable CPU host; not every GPU instance type does.
An attestation verifier in the deployment path
Decide up front whether you will call the hardware or cloud vendor's own attestation service directly (AMD's Key Distribution Service for VCEK certificates, Microsoft Azure Attestation for guest attestation, NVIDIA's Remote Attestation Service for GPU quotes) or route through a third-party attestation broker such as Intel Trust Authority. This choice determines what your inference server's startup code needs to call before it will accept real traffic.
A secrets or key release policy tied to attestation
Confidential computing only pays off if decryption keys for the model weights, or credentials for pulling them, are released only after a passing attestation, not baked into the VM image or environment variables at boot. Have a KMS or secrets manager ready that supports conditional key release, or plan to build that gate yourself before writing any inference code.
An inference stack that tolerates the isolation constraints
Confidential VMs restrict live migration, and enabling confidential OS disk encryption is a decision made at instance creation that cannot be changed afterward. Debugging tools that require host-level introspection, interactive shell access to a GPU driver stack from outside the VM, or certain profiling tools may be restricted or unavailable inside the confidential boundary. Confirm your model-serving framework (vLLM, TensorRT-LLM, Triton, or similar) has been validated to run inside a CC-enabled confidential VM before committing a production rollout to it.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
Deployment Procedure
The steps below describe the Azure path (AMD SEV-SNP CPU TEE plus H100 in CC mode, the combination Azure shipped first to general availability), since it is currently the most complete path to a GPU-capable confidential deployment. The attestation mechanics for an Intel TDX-based confidential VM follow the same shape, substituting Intel's PCS/quoting-enclave chain for AMD's KDS/VCEK chain described in Step 4.
Step 1: Provision the confidential VM size, not a standard GPU instance
Select a confidential-computing GPU VM size at creation time (Azure's NCCadsH100v5 series pairs AMD SEV-SNP with H100). This is a distinct VM size family, not a flag applied to a standard NC-series instance, and the confidential OS disk encryption setting chosen here cannot be changed after deployment, so decide before provisioning whether you need it.
Step 2: Enable boot integrity monitoring and the guest attestation extension
The VM needs Secure Boot and a virtual TPM enabled before the attestation extension can be installed; both are prerequisites, not optional add-ons, for the guest attestation path. Enable integrity monitoring at creation (or on an existing VM by updating it) with the integrity-monitoring flag, then install the guest attestation package from Microsoft's package repository onto the guest OS.
Step 3: Verify CC mode is actually active on the GPU before trusting it
Do not assume CC mode is on because the instance type name implies it. Run nvidia-smi and check for a confidential compute status field reading a protected state; if it reads off or the field is absent, the GPU is not running as a confidential device regardless of what VM size you provisioned, and no data should be sent to it until this is corrected.
Step 4: Generate and verify the CPU-side hardware attestation report
Request an AMD SEV-SNP attestation report from the processor. That report is signed by a Versioned Chip Endorsement Key (VCEK) unique to the specific chip and its current TCB (trusted computing base) version. Fetch the matching VCEK certificate from AMD's Key Distribution Service, chain it back through AMD's SEV CA to AMD's root CA, and verify the report's launch measurement (a hash of initial guest memory and vCPU state) matches the expected value for your known-good VM image before proceeding. Tooling such as the open-source snpguest utility automates report generation, certificate retrieval, and signature verification; do not hand-roll this chain unless you have a specific reason to.
Step 5: Submit the CPU report to Microsoft Azure Attestation and get a signed token
The guest attestation library on the VM reads the AMD SEV-SNP report from the hardware and submits it to Microsoft Azure Attestation (MAA), which evaluates it against your organization's attestation policy and returns a signed JSON Web Token if the VM meets it. This token, not the raw hardware report, is what your key-release logic should require before it will hand over any secret.
Step 6: Generate and verify the GPU attestation quote separately
GPU confidentiality is attested independently of the CPU TEE. Query the GPU for its attestation quote using NVIDIA's attestation SDK, then verify it against NVIDIA's Remote Attestation Service (NRAS), which checks the device certificate and returns a pass/fail result plus a nonce-bound response to prevent replay of an old attestation. Both the CPU (MAA) and GPU (NRAS) attestations need to pass independently; a valid CPU attestation does not imply the GPU is in a valid CC state, and vice versa.
Step 7: Gate key release on both attestation results, then load the model
Only after both the MAA token and the NRAS attestation result come back valid should your startup sequence request the decryption key or credential needed to pull model weights from storage. Load the model into the now-attested confidential VM and GPU, and start the inference server (vLLM, TensorRT-LLM, or equivalent) only after weights are loaded, never before, so that no window exists where the server accepts external traffic on an unverified host.
Step 8: Re-attest on a schedule, not just at boot
A one-time attestation at instance launch only proves the state at that moment. Firmware updates, TCB version rotations from AMD or NVIDIA, or a live-patched host can change the trust state after boot. Build periodic re-attestation into the deployment (a scheduled job that re-runs Steps 4 through 6 and revokes the running key if either check fails) rather than treating attestation as a one-time gate.
Validation: How to Confirm the Deployment Is Actually Protecting Anything
A confidential VM that boots successfully is not proof that confidentiality is actually engaged; several of the steps above fail silently into a less-protected state if skipped. Validate explicitly rather than assuming: confirm nvidia-smi reports the GPU's confidential compute field as protected, not merely that the instance type name contains the letters CC; confirm the MAA token returned in Step 5 is a fresh, policy-passing token by checking its issued-at claim and expiry rather than caching a token from an earlier deployment; confirm the NRAS response in Step 6 is bound to a nonce generated for this specific attestation request, since a cached or replayed attestation response defeats the anti-replay protection entirely; and confirm your key-release logic actually branches on attestation failure in a test where you deliberately submit an invalid or stale report, rather than only testing the happy path. A deployment that has never been tested against a failing attestation has not validated that the gate exists, only that the happy path works.
Failure Modes
These are the specific ways confidential-computing inference deployments break in production, distinct from ordinary infrastructure failures.
CC mode silently off on the GPU
Provisioning the right instance size does not guarantee the GPU firmware has CC mode enabled; a misconfigured image or a host-level setting can leave a nominally confidential-capable GPU running in its default non-confidential state. This fails silently: inference still works, weights still load, and nothing errors, but the memory-encryption guarantee is simply absent. This is why Step 3's explicit nvidia-smi check has to be part of every deployment, not a one-time sanity check during initial setup.
TCB version drift breaking attestation without warning
AMD and Intel periodically issue TCB (trusted computing base) recovery updates in response to disclosed vulnerabilities in the processor's security microcode. When that happens, previously valid attestation reports can start failing verification against an updated policy, or conversely, an un-patched host can start failing a tightened policy. Teams that treat attestation policy as a set-and-forget config discover this only when inference servers start failing to start after a routine AMD or Intel security update, with an error that traces back to a TCB mismatch rather than anything in the model-serving stack itself.
Attestation service outage as a hard dependency
Gating key release on a live call to MAA or NRAS means an outage or rate-limit on those services becomes an outage for the inference deployment's ability to start new instances or rotate keys, even if the underlying hardware is functioning normally. Plan for this dependency explicitly (cached tokens with defined, short expiry windows; alerting on attestation-service latency) rather than discovering it during an incident.
Debugging tooling that assumes host-level access
Standard GPU profiling and driver debugging tools frequently assume the operator can inspect host-level state that a confidential VM specifically walls off. Teams accustomed to attaching a profiler or debugger to a GPU process from outside the VM find those tools blocked or degraded inside the confidential boundary, which is by design, but which needs to be planned for in the operational runbook rather than discovered during an incident response.
Live migration and elasticity constraints
Confidential VMs restrict or disable live migration in ways standard VMs do not, and the confidential OS disk encryption setting is fixed at creation time. Autoscaling groups, blue-green deployment patterns, and maintenance-window host migrations built around assumptions from standard cloud VMs need to be re-validated against the confidential VM's actual operational constraints before relying on them in production.
Security Tradeoffs to Set Expectations On
Confidential computing for AI inference closes a real and specific gap, but it is not a blanket security upgrade, and every team deploying it should be explicit internally about what it does not cover. It protects memory confidentiality and integrity against a privileged host operator or physical attacker with access to the machine; it does not validate that the model weights being loaded are themselves free of a supply chain compromise (a backdoored or tampered model artifact loaded into a perfectly attested TEE is still a backdoored model, faithfully protected). It does not replace network-layer access control, authentication on the inference API, or input validation against prompt injection; those all sit outside the TEE boundary and need their own controls. NVIDIA's own published figures put the GPU-side CC mode overhead at roughly 2 to 5 percent throughput on most LLM inference workloads, plus a one-time attestation cost on the order of 1 to 3 seconds per instance provisioning event, which is a real but generally tolerable cost for most serving patterns; teams should still benchmark on their own model and batch-size profile rather than assuming the published range applies unchanged. Vendors have not published formal detection rates or breach-prevention statistics for confidential computing deployments, because the guarantee is a cryptographic and hardware isolation property, not a detection control, so do not expect or fabricate a percentage where none exists.
The bottom line
Deploying confidential computing for AI inference is a real, achievable procedure with a hardware-backed TEE at the CPU layer (AMD SEV-SNP or Intel TDX), a CC-capable GPU (H100 or H200 in confidential mode), and an attestation chain that gates key release rather than trusting the VM at boot. The work that actually determines whether the deployment protects anything is in the details: confirming CC mode is genuinely active rather than assumed, verifying both the CPU and GPU attestation chains independently rather than treating one as a proxy for the other, re-attesting on a schedule instead of only at launch, and being honest with stakeholders about what the TEE boundary does not cover, including model supply chain integrity, API-level access control, and prompt injection defenses. Teams that build the attestation gate as a first-class, tested part of the deployment rather than a checkbox get the actual security property; teams that provision the right instance type and stop there frequently do not, and have no alerting in place to tell them.
Frequently asked questions
What is confidential computing in the context of AI model inference?
It is running the inference workload inside a hardware-backed trusted execution environment, typically a CPU TEE like AMD SEV-SNP or Intel TDX paired with a GPU in confidential computing mode such as NVIDIA H100 or H200, so that model weights and inference inputs remain encrypted in memory even against a privileged cloud operator or physical attacker with access to the host.
Do I need both a CPU TEE and a confidential GPU, or is one enough?
For GPU-accelerated inference you need both. NVIDIA's confidential computing mode on H100 and H200 extends the trust boundary across the PCIe bus into the GPU, but it depends on a compatible host CPU TEE (AMD SEV-SNP or Intel TDX) underneath it to establish the confidential VM the GPU attaches to; neither one alone protects the full inference pipeline.
Is Intel SGX still the right technology for confidential AI workloads, or should I use TDX?
Intel TDX is the current technology for VM-level confidential computing and is what modern confidential VM offerings use for lift-and-shift workloads like inference servers. SGX provides fine-grained, application-level enclave isolation with tighter memory constraints and is not the technology behind current confidential VM deployments for AI inference; verify which one a given cloud's confidential VM series actually uses before assuming compatibility.
What does attestation actually verify before an inference server can access sensitive data?
Attestation verifies a cryptographically signed hardware report proving the VM and GPU are running the expected, unmodified software in a genuine confidential computing state. On the CPU side this is an AMD SEV-SNP report signed by a chip-unique key and verified through AMD's certificate chain, or the equivalent Intel TDX quote chain; on the GPU side it is a separate quote verified through NVIDIA's Remote Attestation Service. Only after both pass should any decryption key or credential be released to the workload.
What performance overhead should I expect from running inference inside a TEE?
NVIDIA publishes a roughly 2 to 5 percent throughput overhead for confidential computing mode on H100 for most LLM inference workloads, plus a one-time attestation cost on the order of 1 to 3 seconds per instance provisioning event. Actual overhead varies by model, batch size, and workload shape, so teams should benchmark their own deployment rather than assuming the published range applies unchanged.
Does confidential computing protect against a backdoored or tampered model file?
No. Confidential computing protects the confidentiality and integrity of memory while the workload runs; it does not validate the integrity or provenance of the model artifact being loaded into that protected memory. A tampered or backdoored model loaded into a correctly attested TEE is still a tampered model, faithfully protected. Model supply chain integrity requires separate controls such as artifact signing and malware scanning of model files.
Sources & references
- AMD SEV-SNP Attestation (LSS whitepaper)
- AWS: Attest an Amazon EC2 instance with AMD SEV-SNP
- virtee/snpguest (SEV-SNP attestation report tooling)
- Microsoft Learn: What is guest attestation for confidential VMs?
- Microsoft Learn: Azure Attestation overview
- Azure/confidential-computing-cvm-guest-attestation (GitHub)
- NVIDIA Confidential Compute on Hopper H100 (whitepaper)
- NVIDIA Docs: GPU and switch attestation
- NVIDIA Docs: Hopper Single GPU Attestation Example
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.