This article is a technical explanation and implementation example created using AI. The code and procedures presented are based on primary sources, but have not been verified on actual hardware by the author. Operation may vary depending on the environment and version.
In next-generation AI infrastructure, supporting large-scale models and complex inference workloads requires not only computing power, but also optimization of memory bandwidth, die area, and power efficiency. NVIDIA NVLink Fusion and the custom HBM-based die technology NVHBM, announced by NVIDIA, serve as key foundational technologies enabling hyperscalers and AI-native enterprises to integrate their own custom AI accelerators (XPUs) and CPUs into NVIDIA’s AI infrastructure platform. This article outlines the structural features and overall platform benefits brought by these technologies, based on primary information from the official blog.
Challenges in AI Factories and Accelerator Design
As AI systems scale from individual accelerators to rack-level compute domains, accelerator packages must balance compute logic, power delivery, thermal design power (TDP), and high-bandwidth memory. For training, inference, and agentic AI workloads, high-throughput access to model weights, KV caches, and activation data heavily dictates performance.
However, in developing custom accelerators, selecting state-of-the-art memory technology, package integration, and verification can become bottlenecks. NVLink Fusion and NVHBM aim to ease these integration and verification hurdles by providing base dies designed and verified in collaboration with major memory vendors, thereby shortening the time to market for semi-custom AI factories.
flowchart TD
A["カスタム XPU / CPU"] -->|NVLink Fusion & NVLink-C2C| B["NVIDIA AIインフラストラクチャプラットフォーム"]
C[NVHBM] -->|3D積層・カスタムPHY| A
B -->|スケールアップ・スケールアウト・MGXアーキテクチャ| D["データセンター / AIファクトリー"]
Connectivity and Scale-Up with NVLink Fusion
NVLink Fusion is a connectivity technology and IP that allows hyperscalers and AI-native enterprises to connect their proprietary XPUs and CPUs to NVIDIA’s AI infrastructure platform. By leveraging NVIDIA’s scale-up and scale-out technology stack and MGX rack-scale architecture, it reduces the complexity of development and deployment.
The latest 6th-generation NVLink is a scale-up networking fabric for AI factories, enabling high-speed synchronization of activations and hidden states between experts located across different GPUs, even when utilizing advanced routing techniques such as expert parallelism (EP) and WideEP. Furthermore, upstream, XPUs and CPUs can be connected via NVLink-C2C. This enables heterogeneous computing where custom XPUs coexist with GPUs on a single architecture, allowing flexible reprovisioning of datacenter capacity.
Three Platform Benefits Brought by NVHBM
At the package level, NVHBM complements the integrated architecture. NVHBM brings three primary platform advantages to AI accelerator programs: increased memory bandwidth, die area savings, and reduced power consumption.
1. Increased Memory Bandwidth
The performance of an AI accelerator depends on how consistently data can be fed to the compute engines. NVHBM delivers up to 30% more memory bandwidth per stack compared to standard HBM4e. In memory-bound or partially memory-bound AI workloads, this expansion in bandwidth directly translates to improved accelerator utilization and higher throughput. Particularly during large-scale model inference, faster data movement between HBM and compute cores is expected to boost token throughput per user, while preventing local compute engines from starving for data.
2. Die Area Efficiency and Flexibility
In custom AI silicon, area allocation on the package is critical. Standard HBM relies on wider interface connections, which strains the overall package footprint. In contrast, NVHBM adopts a custom base die designed for efficiency by moving the memory controller inside the 3D HBM stack and integrating a custom PHY.
Compared to the JEDEC HBM4e standard, this design reduces PHY and support areas by up to 67%. The narrowed interface also simplifies interposer routing, providing up to 80% more usable silicon area across the layout. The area saved from the reduced memory interface requirement can be used to expand the central AI compute die, achieving up to a 30% increase in main die silicon area for compute and other functions. Designers gain the flexibility to optimize XPU features according to inference serving, recommendation systems, multimodal pipelines, and internal training workloads.
3. Power Efficiency and Scaling
Power is one of the most severe constraints in modern AI infrastructure. HBM power consumption impacts the overall accelerator power budget, package thermal design, rack power envelope, and datacenter cooling plans. NVHBM reduces HBM power consumption by up to 15% compared to standard HBM4e, creating power and thermal headroom for computation.
This power reduction contributes not only to improved performance-per-watt at the individual XPU level, but also has a major impact at the rack and datacenter scale. According to primary sources, in a 1-gigawatt datacenter using 2,000W XPUs, saving power across thousands of accelerators can generate compute headroom to run up to 15,000 additional XPUs. This is especially crucial for large-scale model inference, which repeatedly reads model weights and KV caches with low latency and high throughput.
Overall Performance Impact
According to primary sources, the combination of NVLink Fusion and NVHBM delivers up to a 30% overall end-to-end performance improvement per XPU through rack-scale synergies in increased bandwidth, die area efficiency, and HBM power reduction. Integrating chip-level efficiency with rack-level connectivity provides a more direct path from custom AI accelerator design to production deployment.
Usage Precautions and Future Outlook
The contents explained in this article are based on official information published on the NVIDIA Technical Blog. Actual hardware design and deployment require collaboration status with respective memory vendors and validation according to the characteristics of the targeted AI workloads. Direct verification code using APIs or GUIs [planned for confirmation in a Windows environment] has been omitted from this article as it depends on real-world environment constraints and public SDK specifications. For detailed trends in next-generation custom AI silicon and infrastructure building, please refer to official technical documentation.
コメント