Generative AI has made models the focal point of infrastructure discussions. Agentic AI changes this landscape: as agents retrieve data, call tools, execute code, and coordinate multi-step workflows, the software running around the model becomes just as important as the inference itself.
To meet evolving infrastructure needs, Arm has launched the Arm AGI CPU, built on the Arm Neoverse platform and designed specifically for the computational demands of agentic AI. The Arm AGI CPU aims to provide the performance, scalability, and energy efficiency required for high-concurrency agent workloads, covering orchestration, tool execution, data retrieval, and application services.
At the 2026 Open Compute Project (OCP) Global Summit, Arm introduced a new CPU evaluation methodology for agentic AI, gaining support from Fujitsu, Meta, and OpenAI. Meanwhile, third-party validation results indicate that the Arm AGI CPU can deliver up to twice the performance per rack compared to x86 server processors.
Achieving large-scale deployment of agentic AI requires more than just CPU innovation; it needs a robust software ecosystem covering agent frameworks, orchestration, data infrastructure, networking, and application services. Leveraging years of deep collaboration with the open-source community, software vendors, and cloud service providers, Arm has built a mature, production-validated software ecosystem that lays the foundation for entering this new era. Today, cloud-native and AI software widely supports Arm64 natively, and CPUs based on the Arm Neoverse platform are deployed at scale on leading global cloud platforms. Built on this mature software foundation, developers can construct and scale agentic AI applications on Arm AGI CPUs.
The Software Behind Agentic Tasks
The Agentic AI software stack covers agent frameworks and orchestration, tool execution, memory and retrieval, model serving, and cloud-native infrastructure. As shown below, these capabilities are supported by a broad ecosystem of open-source and commercial technologies, including LangGraph, LangChain, Firecracker, Redis, Elasticsearch, and vLLM.
The Agentic AI software stack integrates agent frameworks, orchestration, runtime services, data retrieval, and model serving onto a unified cloud-native foundation, supporting various applications such as software engineering, robotics, and automation.
Developers can learn about software availability, supported versions, and related resources through the Arm Software Ecosystem Dashboard, which features a dedicated Agentic AI category. This serves as a starting point for developers to build and deploy agentic AI applications on Arm64 using familiar technologies.
Different Agents, Different Infrastructure Requirements
Although various agentic AI applications share a common software foundation, their infrastructure requirements differ significantly based on the tasks they perform. Coding agents require frequent code execution, software compilation, and testing, placing higher demands on CPU performance and sandbox environments; enterprise workflow agents rely heavily on API interactions, database transactions, and orchestration; research agents depend on efficient search, information retrieval, and memory management. Personal assistants prioritize interactive responsiveness, while robotic applications have greater needs for simulation and low-latency coordination.
These workload differences are part of a broader shift in how AI applications are designed and deployed. Learn more about how agentic AI is driving computing developments in the cloud, at the edge, and in physical systems.
For developers, understanding where execution time is primarily spent is key to optimizing the software stack. For example, improving sandbox performance for coding agents, enhancing database and API efficiency in enterprise workflows, and optimizing retrieval and caching for research agents all contribute to better end-to-end task performance. These differences highlight that when evaluating agentic AI infrastructure, one should consider the entire workflow rather than focusing solely on model inference performance.
From Software Adaptation to Performance Optimization
To address these diverse workload demands, Arm is committed to optimizing the performance of key components in the agentic AI software stack, covering both open-source and commercial software.
Tool Execution and Sandboxing: Independent tests by Signal65 on the Arm AGI CPU showed up to a 45% reduction in sandbox creation latency compared to the AMD EPYC 9965, highlighting the importance of CPU performance for agent tool execution.
Enterprise Search and Retrieval: Agents rely on efficient access to structured and unstructured data within enterprises. Elasticsearch supports such workloads through capabilities like vector search, hybrid retrieval, and real-time search. Arm's recently released Elasticsearch performance benchmarks demonstrate performance advantages when running such workloads on Arm-based infrastructure.
Vector Databases and Agent Memory: Vector databases enable agents to retrieve relevant context and access persistently stored knowledge across multi-step workflows. Technologies like Redis support vector search, caching, and agent state management. Arm is working with ecosystem partners to optimize these workloads on Arm64. Redis 8.10.1 vector search benchmarks show that AWS Graviton5 delivers 16% to 27% more context retrievals per dollar compared to x86: under normal load, AWS Graviton5 offers a 16% price-performance advantage on a dataset of 570,000 documents; under high load, this advantage increases to 27% on a dataset of 3.3 million documents.
Model Serving: Model serving frameworks are also a key focus area for Arm software optimization. Arm has released relevant vLLM developer resources demonstrating how to deploy models, perform quantized inference, and tune performance on Arm-based servers, helping developers improve inference throughput and resource utilization efficiency.
Emerging independent software vendors (ISVs) like OpenInfer are driving intelligent inference routing and scheduling for heterogeneous compute resources. OpenInfer is collaborating with Arm to demonstrate how its Loom platform runs AI models across multiple Arm-based CPUs.
Agentic AI in Practice: Customer Service Demo on Arm AGI CPU
To demonstrate how the agentic AI software ecosystem works in practice, Arm developed a multi-agent customer service application running on CPU using LangGraph and LangChain. The application uses Small Language Models (SLMs) hosted on the CPU to classify incoming customer requests and route them to specialized agents responsible for meeting scheduling, handling complaints, product inquiries, and refunds. These agents leverage CPU inference, enterprise data retrieval, external model APIs, and tool execution capabilities to complete a series of multi-step workflows, including scheduling meetings, generating personalized product brochures, and initiating refund processing through enterprise systems. This demo shows how developers can integrate familiar open-source software components to build end-to-end agentic AI applications on Arm-based infrastructure.
For the customer service demo video, please check Arm's official WeChat account.
Building and Optimizing Agentic AI on Arm Platforms
Arm provides developers with a suite of practical resources to help them build, deploy, and optimize agentic AI applications by leveraging the mature Arm64 software ecosystem. Through Arm Learning Paths, developers can access hands-on tutorials to learn how to build AI agents using llama.cpp, implement RAG applications with LangChain and FAISS, and deploy vLLM model serving infrastructure on Arm-based platforms. These resources enable developers to move beyond basic software adaptation toward building and optimizing real-world agentic AI applications.
The growing ecosystem of AI agent skills also makes it easier for developers to build, migrate, and optimize applications on Arm-based infrastructure. AWS has launched Graviton developer skills to help developers use AWS Graviton; meanwhile, Arm has contributed its own skills to the OpenAnolis community's SkillHub, including Arm Performix, Arm Performix CI, and Arm Migrate Ease. These initiatives collectively integrate platform-specific knowledge and optimization guidance into AI-assisted development workflows.
Arm continues to collaborate with the open-source community, software partners, and cloud service providers to drive the development of the agentic AI software ecosystem. Developers can view supported frameworks and technologies via the Arm Software Ecosystem Dashboard and start developing on Arm64 platforms immediately using Arm Learning Paths. Through Arm Learning Paths, developers can also gain practical guidance via hands-on tutorials on how to:
· Build AI agents using llama.cpp and llama-cpp-agent;
· Execute AI-generated code within isolated Firecracker microVMs;
· Implement RAG applications using LangChain and FAISS;
· Deploy vLLM model serving infrastructure on Arm-based platforms.
These resources help developers progress from basic software adaptation to building and optimizing real-world agentic AI applications.
– End –