Artificial Intelligence models are evolving from simple responders into systems that reason through multi step tasks, invoke tools, and issue follow ups. That shift is pushing inference to the center of compute budgets. A new independent benchmark, InferenceMAX v1 from SemiAnalysis, measures the total cost of compute across real world scenarios and finds Nvidia’s Blackwell platform leading on both performance and efficiency for large scale operations. The analysis even points to a potential 15x return on investment for a GB200 NVL72 class system, underscoring how inference economics are reshaping infrastructure strategy. As Ian Buck, Nvidia’s vice president of hyperscale and high performance computing, put it, inference is where value is delivered every day.
InferenceMAX arrives as generative workloads produce far more tokens per query, making efficiency a competitive advantage. The study highlights Nvidia’s deep work with the open source community, citing collaborations with OpenAI on gpt oss 120B, Meta’s Llama 3 70B, and DeepSeek AI’s DeepSeek R1. Partnerships with the developers of FlashInfer, SGLang, and vLLM have yielded kernel and runtime improvements that push open models to new speeds while keeping results reproducible and transparent.
Software remains a major unlock. Nvidia’s TensorRT LLM library on DGX Blackwell B200 systems has already stretched open source large language models, and the TensorRT LLM v1.0 update adds smarter parallelization while tapping NVLink Switch’s 1,800 GB per second fabric to raise throughput. The gpt oss 120b Eagle3 v2 model introduces speculative decoding to predict multiple tokens at once, cutting latency and enabling up to 30,000 tokens per GPU, five times more than before. Dense models like Llama 3.3 70B also benefit, surpassing 10,000 tokens per second per GPU on B200 hardware, a fourfold jump over the H200 generation.
Cost metrics now matter as much as raw speed. Tokens per watt and cost per million tokens determine margins at data center scale. On these measures, Blackwell stands out, delivering 10x more throughput per megawatt and a 15x reduction in cost per million tokens compared to the prior generation. Using a Pareto frontier to weigh throughput, energy, and responsiveness, InferenceMAX shows Blackwell consistently sitting on the efficient edge rather than trading one metric for another.
Under the hood, Blackwell’s advantage is a tightly coupled hardware software design. NVFP4 precision boosts efficiency without giving up accuracy, while fifth generation NVLink interconnects up to 72 GPUs so they operate like a single processor. NVLink Switch orchestrates parallel workloads across tensors, experts, and data streams to maintain high concurrency. Nvidia says post launch optimizations have already doubled Blackwell performance, powered by open frameworks such as TensorRT LLM, TensorRT LLM, Nvidia Dynamo, SGLang, and vLLM, and a broad ecosystem of more than 7 million CUDA developers contributing to over 1,000 open source projects.
The industry is moving from pilots to Artificial Intelligence factories that turn data into tokens, predictions, and decisions in real time. Open, transparent benchmarks like InferenceMAX help teams select hardware, contain costs, and plan for service levels as demand rises. Nvidia’s Think SMART framework targets this transition, where inference performance is inseparable from financial outcomes. In today’s Artificial Intelligence inference race, speed is crucial, but efficiency decides who stays ahead.
