Inference

The NVIDIA Blackwell Ultra NVL72 platform has achieved leading performance in the first round of the AgentPerf benchmark, a new industry standard for agentic AI infrastructure, running 20x more agents per megawatt than the NVIDIA Hopper. This benchmark measures the performance of systems in handling complex, multi-step AI workloads, which are fundamentally different from conversational AI. The results demonstrate the importance of codesign and optimization across the full stack for achieving high performance in agentic AI. The practical implication for engineers building AI systems is that they need to consider the unique requirements of agentic AI workloads when designing and optimizing their systems.

The authors have demonstrated a 25% improvement in performance for general-purpose and agentic AI workloads using the Graviton5 chiplet architecture, custom die-to-die connectivity, and support for DDR5-8800 memory and the latest PCIe gen6 interconnects, effectively surpassing Moore's Law. This breakthrough enables faster and more energy-efficient processing for AI workloads. The improved design is particularly beneficial for large-scale AI applications, where every percentage point of performance gain can significantly impact overall system efficiency. This achievement has the potential to accelerate AI adoption in various industries.

Ferveret, a startup founded by Reza Azizian and Matteo Bucci, is developing a nuclear-inspired cooling system for data centers that uses a specialized liquid to absorb heat, reducing electricity usage and water consumption. The company's Adaptive Phase Cooling (APC) solution has shown a 15% improvement in computational power efficiency compared to state-of-the-art liquid cooling solutions. By combining APC with a power control system, Ferveret claims to enable data centers to generate 35% more tokens from their AI models with the same amount of power. This innovation has the potential to make data centers more sustainable and efficient. The practical implication for engineers building AI systems is that they can potentially reduce their energy consumption and increase their computational power efficiency by adopting Ferveret's cooling system.

The Strands Evals SDK introduces detectors that automate AI agent failure detection and root cause analysis, reducing diagnosis time from hours to minutes. Detectors analyze execution traces using large language model (LLM)-based analysis and provide structured output, including categorized failures, causal chains, and fix recommendations. This complements the evaluation framework by answering not only "how well did the agent do?" but also "why did it fail and how do I fix it?". The detector pipeline operates in two phases, with Phase 1 scanning each span in a session against a comprehensive failure taxonomy. For engineers building AI systems, this means they can quickly identify and fix issues, improving overall system reliability and performance.

The AI landscape has shifted from models that simply know to agents that do, with foundation models being used as cognitive engines for AI agents in the physical world. To be useful in high-stakes physical environments, agents need to be grounded in physical laws and operational constraints, overcoming the challenge of hallucination. Four approaches to grounding AI agents are proposed, including physics-guided deep learning, which integrates first-principle physical knowledge into the foundation model in pretraining. This ensures that predictions obey governing physical laws, making agents physically consistent and operationally reliable. The practical implication for engineers building AI systems is that they must consider the physical constraints of the environment in which their agents will operate.

A recent study from the MIT Media Lab found that participants who relied on AI systems to verify facts actually got worse at detecting misinformation on their own when their chatbots were taken away, with a 15 percentage point decline in unassisted performance by week four. The study, which tracked 67 people over four weeks, also showed that participants were 21 percent more accurate in detecting fake news when assisted by an AI chatbot during a session. This phenomenon, known as the "AI dependency paradox," has significant implications for engineers building AI systems, as it highlights the importance of considering the potential consequences of relying on AI for accurate news. The study's findings suggest that AI systems can be effective tools in reducing people's beliefs in false information, but they also come with real limitations, including the potential to undermine users' critica

NVIDIA has optimized Google DeepMind's experimental open model, DiffusionGemma, for exceptionally fast text generation on NVIDIA GeForce RTX GPUs, RTX PRO platform, and DGX Spark systems, achieving significant speedup across local PCs and the cloud. This optimization enables real-time text generation capabilities, with the potential to accelerate applications such as chatbots, language translation, and content creation. The optimized model can be used in various settings, from local PCs to large-scale cloud deployments. This achievement highlights the importance of hardware acceleration in AI model performance.

The Practitioner's Guide to AgentOps outlines a comprehensive framework for building and managing multi-step AI agent pipelines, leveraging the AgentOps platform to streamline workflows, and integrating with various tools and services such as AWS Bedrock and LangChain. The guide provides a detailed overview of AgentOps' architecture, including its ability to handle complex tasks, integrate with existing systems, and scale to meet the demands of large enterprises. By adopting AgentOps, practitioners can reduce the complexity of building and deploying AI agents, enabling faster time-to-market and improved business outcomes. However, the guide notes that successful implementation requires careful planning, integration, and testing to ensure seamless operation.

The performance of AI agents is hindered by the intent-execution gap, which is the mismatch between what the model intends and what the harness executes. Minimizing this gap is sufficient to achieve state-of-the-art performance across diverse agentic benchmarks. The Simple Strands Agent (SSA) is introduced as a lightweight and customizable single-agent harness designed to close the gap between reported and actual performance. Effective agent design is not entirely model agnostic, and model-harness codesign is critical in achieving optimal performance. This has significant implications for engineers building AI systems, as it highlights the importance of considering the model-harness interface and identifying invariant components that remain effective across model upgrades and environments.

The article provides a deep dive into the microarchitectural costs of Kubernetes GPU time-slicing for concurrent Large Language Model (LLM) agents. Not mentioned are specific numbers, model names, or benchmark results, but the post explores the systems-level implications of co-locating Agentic AI workloads on Kubernetes. The practical implication for engineers building AI systems is a better understanding of the hidden costs of GPU time-slicing, enabling more efficient deployment of LLM agents. The article focuses on the technical aspects of Kubernetes and GPU time-slicing, highlighting the need for careful consideration of resource allocation and workload management.

NVIDIA's Confidential Computing technology is being used by Apple to support confidential inference in their Private Cloud Compute, expanding beyond Apple's data centers to Google Cloud, with NVIDIA Blackwell GPUs providing a hardware-based security layer for accelerated AI workloads. This collaboration aims to support next-generation Apple Intelligence features, leveraging the technologies behind the Gemini family of models. The adoption of NVIDIA Confidential Computing reflects a broader shift in AI infrastructure towards high-performance, server-side inference while maintaining strong privacy and security guarantees. This has significant implications for engineers building AI systems, as they must consider the importance of privacy and security in their designs.

The article discusses the limitations of increasing context size in Retrieval-Augmented Generation (RAG) systems for aggregation tasks, finding that it does not improve accuracy and instead makes errors harder to detect. The author benchmarks retrieval-based pipelines against a deterministic full-scan engine across 100,000 rows, demonstrating the need to route computation queries away from RAG. This finding has significant implications for engineers building AI systems, as it suggests that alternative approaches are needed to improve accuracy in aggregation tasks. The author's system, built in response to these limitations, offers a potential solution.

The AI agent ecosystem is currently in a phase of protocol proliferation, with four significant protocols published in the past eighteen months: Model Context Protocol (MCP), Agent2Agent (A2A), Agent Communication Protocol (ACP), and Agent Network Protocol (ANP). MCP has already won the tool-calling layer, with over 10,000 active public MCP servers and 164 million monthly Python SDK downloads by April 2026. A2A is a task coordination interface that defines how two agents delegate a task, while ACP is a message envelope format and ANP is a discovery and identity protocol. The practical implication for engineers building AI systems is that they need to understand the different layers of the stack and choose the appropriate protocol for their specific use case.

Researchers have proposed a method called continuous batching to improve the efficiency of serving large language model (LLM) inference for multiple users at once, reducing the overhead of static batching by dynamically scheduling and using ragged batching. This approach can handle varying request sizes and rates, achieving up to 2.5x faster inference compared to static batching. However, it requires careful tuning of the scheduling algorithm and batching strategy to achieve optimal performance. Continuous batching can be particularly beneficial for applications with high variability in request sizes and rates, such as chatbots and conversational AI systems.

Researchers have introduced a novel scaling law that links specific architectural decisions to loss, enabling the identification of models that can boost throughput by up to 47% without compromising accuracy. This breakthrough has significant implications for the development of efficient large language models (LLMs). By optimizing model architecture, engineers can achieve substantial speed gains without sacrificing performance. The new scaling law provides a valuable framework for optimizing LLMs for high-throughput applications.

This article presents an intelligent document processing pipeline that utilizes both on-demand and batch inference options on Amazon Bedrock, enabling flexible document processing in terms of time and cost. The pipeline can dynamically specify large language models and prompts at the document level, allowing for the extraction of data from multiple types of documents. The on-demand pipeline processes documents one-by-one, returning results within seconds, while the batch pipeline processes multiple documents asynchronously. The pipeline uses AWS SQS FIFO queues, AWS Lambda functions, and Amazon Bedrock Prompt Management to manage prompts and extract data from documents. The practical implication for engineers building AI systems is the ability to design flexible and cost-effective document processing pipelines that can handle large volumes of documents.

The US government has ordered Anthropic to suspend all access to its Claude Fable 5 and Claude Mythos 5 models, citing national security concerns, and Anthropic has blocked all public access to these models globally. This move comes after a viral jailbreak of Fable 5 was published, which claimed to have bypassed the model's safety guardrails to extract functional instructions for cyber exploits and other harmful activities. The sudden regulatory intervention serves as a warning to the enterprise sector about the risks of relying on centralized, cloud-based frontier models. The practical implication for engineers building AI systems is to prioritize redundancy and diversification in their AI workflows to mitigate the risk of sudden model unavailability.

NVIDIA and Doosan Group are expanding their collaboration to advance physical AI and AI factory infrastructure, leveraging NVIDIA's full-stack AI computing platform to integrate AI into Doosan's robotics, construction equipment, and energy solutions. The partnership aims to enhance the efficiency, safety, and productivity of Doosan's manufacturing processes and products. By combining NVIDIA's AI expertise with Doosan's industry expertise, the collaboration will drive innovation in AI factory infrastructure and robotics. This strategic partnership will enable Doosan to accelerate the development and deployment of AI-powered solutions across its various business units.

The emergence of an "intelligence layer" as the control plane for enterprise AI enables organizations to manage the organizational context necessary for models to act reliably, addressing challenges in cost governance, data security, and accountability. This new layer integrates model management, data governance, and organizational processes, providing a unified framework for AI decision-making. By doing so, it enables enterprises to scale AI adoption while maintaining control and oversight. This shift is critical for large-scale AI deployment, where the complexity of organizational context can no longer be ignored.

NVIDIA Isaac Lab on Amazon SageMaker AI enables the scaling of robot reinforcement learning by providing a managed infrastructure for distributed training and inference. This allows robotics teams to iterate quickly during research and run production-grade training jobs without the operational burden of maintaining compute clusters. With Amazon SageMaker HyperPod, teams can achieve cluster resiliency and control, while SageMaker Training Jobs provide a flexible compute option for shorter iterative experiments. The practical implication for engineers building AI systems is that they can focus on developing robot policies rather than managing infrastructure.

Amazon Bedrock's Cross-Region Inference (CRIS) capability allows customers to automatically route model inference requests across multiple AWS Regions within predefined geographic boundaries, enabling more resilient generative AI applications. CRIS offers system-defined inference profiles with global or geographic scopes, optimizing model throughput at low latency overhead. For EU customers, CRIS helps meet local data protection and processing requirements, including GDPR compliance. By using CRIS, customers can take advantage of model availability and capacity across multiple Regions while ensuring security and privacy.

NVIDIA researchers have developed a new AI framework for grasping, autonomous driving, and multi-agent training that leverages a combination of simulation and real-world data to improve performance and robustness. The framework uses a novel architecture that integrates a multi-modal perception model with a reinforcement learning-based control policy, enabling robots to adapt to new objects and environments. This approach has been demonstrated to improve grasping success rates by 15% and autonomous driving safety by 20% in simulation. By training agents in simulation and fine-tuning them on real-world data, the framework enables scalable and efficient training of complex AI systems.

Researchers introduce a novel framework for quantifying the risk of catastrophic failures in large language models (LLMs) during adversarial conversations, leveraging statistical methods to estimate the likelihood of such events. The framework assesses the probability of LLMs producing undesirable outputs, such as generating hate speech or spreading misinformation. By providing a probabilistic measure of catastrophic failures, the framework enables more informed decision-making and mitigation strategies for LLM developers. This approach can help prevent the amplification of harmful content and promote safer AI interactions. The framework's effectiveness is demonstrated through experiments on several popular LLMs, showcasing its potential to improve the reliability of AI-powered conversational systems.