This article is a technical explanation and implementation example generated using AI. The code and procedures presented are based on primary sources, but have not been verified on actual hardware by the author. Behavior may vary depending on the environment and version.

Mechanisms and Optimization of Secure AI Inference Using NVIDIA Confidential Computing

When using large language models (LLMs) for enterprise or individual use to securely process highly confidential information and proprietary model contexts, execution in a trusted hardware environment is essential. Based on primary information from the official technical blog, this article organizes the mechanisms of high-performance production AI inference in NVIDIA Confidential Computing (CC) environments, adaptation measures in the TensorRT-LLM inference framework, and performance measurement methodologies.

Purpose and Limitations of This Article

The purpose of this article is to analyze official information to understand how framework-level adaptations are implemented to maintain performance in secure AI inference environments leveraging the NVIDIA Blackwell architecture and Confidential Computing. Implementations and operational verifications on actual hardware are not performed, focusing instead on explaining the components and optimization approaches described in primary sources.

Prerequisites and Cautions

• Execution Environment Prerequisites: The evaluation of primary information utilizes an NVIDIA DGX B200 system (eight B200 GPUs), an Intel TDX platform, and Ubuntu-based host and guest OS environments.

• Limitations Regarding Unverified Hardware: The numerical values and measurement results introduced in this article are based on primary sources and have not been verified on actual hardware by the author.

Components of NVIDIA Confidential Computing

NVIDIA Confidential Computing protects data and workloads in use using hardware-level security features. The main components are as follows.

• Memory Encryption Confidential Virtual Machines (CVM): Encrypts memory per virtual machine to prevent unauthorized access from the host side.

• Confidential GPUs: Protects data and computations processed on the GPU.

• Encrypted NVIDIA NVLink: Encrypts inter-GPU communication to prevent data eavesdropping and tampering along communication paths.

flowchart TD
    A[セキュアなクライアント入力] --> B[CVMメモリ暗号化]
    B --> C[Confidential GPU処理]
    C --> D[暗号化されたNVLink通信]
    D --> E[安全なAI推論出力]

Workload Characteristics and Evaluation Methods for CC Overhead

In a secure execution environment, assumptions regarding memory movement, timing measurements, and multi-GPU communication change, which leads to performance degradation (overhead) if countermeasures are not implemented. Primary sources cite workload characteristics that clearly expose overhead as long input contexts, extended output generation, and low concurrency.

• Evaluation Model: nvidia/DeepSeek-R1-0528-NVFP4

• Inference Framework: TensorRT LLM (PyTorch backend)

• I/O Sequence Length: 32K input / 1K output

• Concurrent Request Count: 1, 2, 4, 8, 16

Comparative evaluations are performed with Confidential Computing disabled (CC off) and enabled (CC on), keeping all other conditions completely fixed.

CC-Aware Adaptations in TensorRT LLM

To maintain performance in a secure environment, TensorRT LLM incorporates several CC-aware adaptations.

1. Adaptation of Host-Device Data Movement

In the B200 Confidential Computing environment, the GPU cannot directly access protected CVM memory, so transfers from the host to the device go through a software-encrypted bounce buffer.

• Host-to-Device: Adapted to select pageable memory instead of pinned memory.

• Device-to-Host: Migrated the readback of repeated tokens and sampling data to asynchronous workers so that the protected copy process does not block the main scheduler.

2. Timing Stabilization for Kernel Autotuners

In standard environments, CUDA events are used to compare candidate tactics, but under CC environments, CUDA event timestamps cause unstable signals. To avoid this, it switches to perform tactic measurements using the GPU's %globaltimer.

3. CC-Compatible Multi-GPU Communication Selection

In the B200 CC configuration, NVLS (NVLink SHARP) multicast cannot be utilized. Therefore, the framework must detect NVLS availability and select the optimal communication algorithm based on message size and topology.

Verification Method (Evaluation Metrics Based on Primary Sources)

Measurement results in the primary source indicate that for a concurrency range of 1 to 16, CC-enabled (CC on) maintained 96.1% to 98.2% of the output token throughput of CC-disabled (CC off), and the average TPOT (Time Per Output Token) overhead remained within 1.2% to 4.3%. These have not been verified on actual hardware and may differ from measured values in an actual environment.

Limitations and Conclusion

Points to Verify and Constraints Prior to Execution

• All contents of this article are based on research and explanations from the official technical blog, and operational verification on actual hardware has not been conducted by the author.

• Operation and the degree of overhead vary depending on the hardware generation, firmware, OS kernel, and driver versions.

• When deploying custom workloads, it is essential to run comparative benchmarks between CC-on and CC-off in the target environment.

In conclusion, when introducing private AI inference using Confidential Computing, it is important to address security configuration and inference optimization as an integrated engineering challenge rather than separating them.

References

• Source Title: Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing

• Source URL: https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing/