Understanding Key Points of GPU Sizing and TCO Optimization in AI Inference

This article is a technical commentary and implementation guide created using AI. Although the code and procedures provided are structured based on primary information, the author has not performed operational verification on actual hardware. Operation may vary depending on the environment and version.

With the widespread adoption of AI, appropriate GPU resource sizing and Total Cost of Ownership (TCO) optimization for inference workloads have become critical challenges. Primary information explains an approach that categorizes inference workloads by use case and builds data-driven footprints based on specific input factors. Based on the official blog, this article organizes its components and optimization methods.

Token Patterns by Use Case and Classification of INF (Inference) Workloads

Inference sizing and TCO optimization begin by clarifying the problem to be solved. Primary information states that the majority of inference workloads fall into the following four categories.

• AI Chatbots/Copilots

• AI Agents (AI agents performing advanced research and reasoning)

• Content Generation

• Translation Apps

These are distinguished by patterns of cached input tokens, input sequence length (ISL), and output sequence length (OSL). For example, chatbots and copilots tend to have long inputs and short outputs (e.g., limited RAG or multi-turn conversations), while AI agents feature extremely long contexts exceeding 128,000 tokens (e.g., deep research or extended RAG). Additionally, content generation is described as having short inputs and long outputs, and translation apps have relatively balanced token lengths.

Balancing Cost and Operations via the Core and Flex Capacity Model

To prevent cost increases due to unpredictable workloads, primary information proposes a strategy called “Core and Flex”.

• Core: Establish a baseline for steady-state workloads using on-premises or reserved cloud GPUs. This suppresses price volatility risk and ensures reliable service for the majority of users.

• Flex: Incorporate public cloud elasticity (spot or on-demand GPUs) for surging traffic, new feature launches, experiments, and more.

This model is said to make it possible to balance capital efficiency (Capex) and operational agility (Opex), preventing over-provisioning or hindering growth.

flowchart TD
    A["Total Workload"] --> B["Core: On-Prem / Reserved GPUs"]
    A --> C["Flex: Spot / On-Demand GPUs"]
    B --> D["Handling Steady-State Traffic"]
    C --> E["Handling Burst / Sudden Traffic"]

Key Input Factors Influencing Sizing

In addition to understanding use cases, sizing plans are built based on the following factors.

• Model Selection (LLM): Larger models are not always optimal; consider mainstream models or fine-tuned smaller models that match your requirements.

• Application Scale and DAU / Concurrency: Understand Daily Active Users (DAU) and the number of concurrent requests issued simultaneously. High concurrency places a significant load on GPU memory and latency.

• Input/Output String Length (ISL / OSL): Longer token lengths per prompt increase GPU memory and computational demand.

• Cache Hit Rate: Estimate the proportion of input tokens reused across requests, processing them from the KV cache to skip prefill and reduce TTFT (Time to First Token) and costs.

• Latency Metrics: Consider average, 99th percentile, and inter-token latencies for responsive TTFT in user experience.

• Contract Duration: Long-term contracts or on-premises are suitable for predictable traffic, while flexible cloud capacity is suited for variable workloads.

Workload-Specific Scenarios and Recommended Memory

Primary information lists four scenarios as specific enterprise use cases.

1. Financial Services (Copilot for Relationship Managers):

   • Characteristics: Analysis of complex client emails (long input, short output).

   • Requirements: Sub-1-second TTFT, 10-50 concurrent sessions, high precision (FP16/BF16), medium-scale model with 7-13B parameters.

   • Memory Recommendation: Approx. 24GB for 7-8B models, approx. 48GB for 13B models.

2. Life Sciences (AI Agents for Drug Discovery):

   • Characteristics: Processing full-text scientific papers (extremely long context).

   • Requirements: TTFT under 2 seconds, 20-30 concurrent users, high precision, long-context model supporting 16K-32K tokens.

   • Memory Recommendation: Very high memory capacity exceeding 80GB per unit.

3. Media & Marketing (Real-Time Content Generation):

   • Characteristics: Generating personalized emails and ad copy from short briefs.

   • Requirements: Sub-1-second TTFT, 50-100+ concurrent users during campaigns, FP16, general instruction-tuned 3-7B model.

   • Memory Recommendation: 16-24GB per GPU.

4. Tech Consulting (Large-Scale Translation Platform):

   • Characteristics: Code and document translation for global teams (1,000 tokens input/output each).

   • Requirements: Low TTFT, hundreds of concurrent requests, FP16 or INT8, medium-to-large multi-language and code-capable model.

   • Mem Recommendation: Entry-level 8-16GB GPUs (suitable for operation in distributed cloud environments).

Model Optimization Methods for TCO Optimization

Strategically reducing the model’s memory footprint is an extremely effective approach when optimizing TCO. Primary information lists three levers in ascending order of effort.

1. Quantization

By lowering numerical precision (e.g., from FP16 to FP8/INT8), memory is reduced by 25-50% without retraining. Reducing floating-point arithmetic precision from 16 bits to 8 bits (1 byte) cuts weight memory by approximately half, enabling the adoption of smaller GPUs, larger batch sizes, and expanded KV caches. Primary information provides code using the NVIDIA Model Optimizer as an example of Post-Training Quantization (PTQ).

import torch
import modelopt.torch.quantization as mtq
from modelopt.torch.export import export_hf_checkpoint
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct", dtype=torch.float16, device_map="auto"
).eval()
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

def calibration_loop(model):
    for prompt in ["Summarize this client email:", "What are the key risks here?"]:
        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
        with torch.no_grad():
            model(**inputs)

model = mtq.quantize(model, mtq.FP8_DEFAULT_CFG, forward_loop=calibration_loop)
export_hf_checkpoint(model, export_dir="./llama-3.1-8b-fp8")

It is shown that this FP8 post-training quantization reduces the weight memory of Llama-3.1-8B from 16.06GB to 9.08GB (a 43.5% reduction) without retraining. FP8 is nearly lossless in inference and has wider headroom than INT8 or INT4, making it the recommended starting point.

2. Pruning and Knowledge Distillation

When quantization alone is insufficient, pruning (depth-wise and width-wise pruning) is performed to remove less important layers or neurons. Furthermore, knowledge distillation is combined to recover accuracy by training the pruned “student model” against the original “teacher model.” This achieves sustainable cost reductions in hardware utilization efficiency, power, and operational overhead.

References

• How to Size GPUs for AI Inference and TCO Without Overspending