Inspur Builds Agent Infrastructure for 40,000 Agents per Rack and Multi-Model Collaboration
AI infrastructure is moving beyond the era of serving one model request at a time. A production agent does more than accept a prompt and return an answer. It may break a task into stages, call external tools, maintain context, coordinate with sub-agents, review intermediate outputs, and remain active for long periods. When an enterprise deploys thousands of these agents at once, the infrastructure requirements look very different from ordinary chatbot inference. At the 2026 Open Compute Technolo
Inspur Builds Agent Infrastructure for 40,000 Agents per Rack and Multi-Model Collaboration
Introduction
AI infrastructure is moving beyond the era of serving one model request at a time.
A production agent does more than accept a prompt and return an answer. It may break a task into stages, call external tools, maintain context, coordinate with sub-agents, review intermediate outputs, and remain active for long periods. When an enterprise deploys thousands of these agents at once, the infrastructure requirements look very different from ordinary chatbot inference.
At the 2026 Open Compute Technology Summit in Beijing, Inspur Information introduced two infrastructure directions for this new workload:
- A CPU-native liquid-cooled rack server designed to support more than 40,000 concurrent agents in one rack.
- A combination of the MetaBrain SD200 supernode and EPAI multi-model fusion API, allowing several large models to generate, review, and combine answers for complex tasks.
The first direction focuses on scale: keeping large numbers of long-running agents online. The second focuses on quality: allowing multiple models with different strengths to work together instead of forcing one model to handle every part of a difficult task.
Agent Workloads Create a Different Infrastructure Problem
Traditional large-model inference often follows a simple pattern:
- A user sends one request.
- The model performs inference.
- The system returns one response.
- The session ends or waits for another request.
An agent application has a much longer execution path.
A single business task may involve:
- Planning
- Task decomposition
- Retrieval
- Tool calls
- Code execution
- Permission checks
- Sandboxed processes
- Multi-turn reasoning
- Sub-agent coordination
- Output validation
- Result aggregation
- Long-running state management
The infrastructure is therefore supporting not only model inference but also a large population of persistent software processes.
In an enterprise environment, the number of active agents may rise from dozens to thousands or tens of thousands. Some agents may run continuously, while others are created dynamically for short tasks and destroyed after completion.
This changes the balance between CPU and GPU resources.
GPUs Generate Tokens
GPUs remain essential for:
- Large-model inference
- Prefill and decoding
- Embedding generation
- Multimodal processing
- High-throughput token production
- Model-parallel execution
CPUs Run the Agent Environment
CPUs handle much of the surrounding work:
- Agent process scheduling
- Tool invocation
- Context and state management
- Business-system interaction
- Sandbox execution
- Network and storage operations
- Authentication and permissions
- Workflow coordination
- Result preparation
A model may produce the reasoning or text, but the CPU often runs the operating environment in which the agent acts.
As a result, agent infrastructure is moving from a GPU-centered design toward a coordinated system involving CPUs, GPUs, networking, storage, cooling, and orchestration software.
Why CPU Density Is Becoming Important
In ordinary enterprise servers, CPU density has historically been limited by power, cooling, space, cables, fans, and maintenance requirements.
Agent deployments change the economics.
If thousands of agents need CPU resources for orchestration, tools, and isolated runtime environments, low-density CPU racks require more floor space, more network connections, and more supporting infrastructure.
At the same time, AI data centers are moving toward much higher rack power.
According to the source report, Inspur Information expects domestic AI racks to approach 300 kW, while some global designs are already moving toward megawatt-class rack systems. Traditional air cooling, commonly limited to several tens of kilowatts per rack, becomes increasingly difficult at those densities.
This is why liquid cooling is no longer only a GPU issue.
CPU racks supporting agent workloads must also match the power and cooling architecture of next-generation AI data centers.
One Rack Can Support More Than 40,000 Agents
Inspur introduced what it describes as the industry’s first CPU-native liquid-cooled full-rack server.
The system is based on the liquid-cooled OCM 2.0 architecture and supports both x86 and Arm processors.
Its headline specifications include:
| Feature | Vendor-Reported Specification |
|---|---|
| Maximum CPUs per rack | 384 |
| Supported agent concurrency | More than 40,000 agents |
| Processor architecture | x86 and Arm |
| Cooling scope | CPU, memory, SSD, NIC, optical modules, and other heat-generating components |
| Compute density | Four CPUs in a 0.5U space |
| Maintenance | Liquid-connected lifecycle maintenance |
| Target environment | High-density and gigawatt-scale AI data centers |
The system does not treat liquid cooling as an attachment added after the server has already been designed.
Instead, the compute layout and cooling architecture are developed together.
From CPU Cold Plates to Full-Component Cooling
A conventional cold-plate server may cool the processors directly while still relying on fans for memory, networking, storage, power components, and other devices.
That approach becomes less effective as density increases.
Inspur’s native liquid-cooling design brings the main heat-generating components into the same cooling architecture:
- CPUs
- Memory
- Network adapters
- Optical modules
- SSDs
- Communication components
This reduces dependence on internal airflow and allows components to be arranged more densely.
A Compact Compute Unit
The source report describes a compact compute unit that places multiple processors and surrounding components into a thin form factor.
The design aims to reclaim space that would otherwise be occupied by:
- Large fans
- Air channels
- Cooling pipes
- Internal cables
- Traditional server trays
A flatter component layout allows one large cold plate to cover more of the system.
The rack also uses a cable-reduced or cable-free internal design and supports maintenance without interrupting the entire service. Inspur says this improves full-rack deployment and maintenance efficiency.
Why OCM Matters
OCM stands for Open Compute Module.
The modular approach is intended to separate the processor module from the wider system design. This can make it easier to support different CPU generations or architectures without redesigning the whole rack around every processor.
For enterprises and data-center operators, that can offer several benefits:
- More hardware choice
- Easier platform upgrades
- Reduced redesign work
- Better compatibility across x86 and Arm
- A longer lifecycle for the surrounding rack infrastructure
The practical value depends on ecosystem adoption, interoperability, and the availability of compatible components.
Scale Alone Does Not Make an Agent Smarter
Running tens of thousands of agents solves a capacity problem. It does not automatically improve the quality of their answers.
Large models have different strengths.
One may be better at:
- Logical reasoning
- Coding
- Long-context analysis
- Technical research
- Natural writing
- Structured extraction
- Multilingual tasks
Even very large models remain uneven across domains.
For complex work, one model may overlook an issue that another model catches. A single model can also produce a confident answer without exposing uncertainty or considering alternative interpretations.
Inspur’s second infrastructure direction is therefore based on multi-model collaboration.
Multiple Models Work on the Same Task
The EPAI multi-model fusion API sends one complex task to several candidate models in parallel.
Each model generates an independent answer. A separate review-and-fusion model then compares the candidates and identifies:
- Shared conclusions
- Conflicting judgments
- Missing information
- Unique insights
- Unsupported statements
- Areas requiring reconciliation
The platform then produces one combined answer.
This is not simple majority voting, and it is not a direct concatenation of every response. The intended workflow is:
- Candidate generation
- Cross-model review
- Difference and omission analysis
- Fusion into a final output
Inspur reports that the system reached 53.9% on the DRACO evaluation, outperforming each individual model in the candidate pool used for that test.
That result should be read as a platform-reported benchmark rather than a universal claim that model fusion always beats the best single model. Performance will depend on the candidate models, evaluator model, routing logic, task type, prompts, and scoring methodology.
Simple Tasks Still Use a Single Model
Running several models for every request would increase cost and latency unnecessarily.
EPAI therefore distinguishes between short, predictable tasks and complex tasks.
Tasks Suitable for One Model
A lightweight model may be enough for:
- Simple question answering
- Format conversion
- Basic extraction
- Classification
- Short tool calls
- Routine content transformation
Tasks More Suitable for Multi-Model Fusion
Several models may be useful for:
- Deep research
- Technical comparisons
- Architecture analysis
- Complex planning
- Long-form reasoning
- High-stakes review
- Tasks with ambiguous evidence
- Outputs that benefit from independent verification
This routing principle is important for production systems.
Multi-model collaboration is most valuable when the potential quality improvement justifies the additional token cost, GPU time, and response latency.
The EPAI Multi-Model Fusion Workflow
The EPAI API is designed to hide much of the multi-model orchestration behind one interface.
A developer sends one request to the platform. EPAI manages:
- Candidate model selection
- Parallel task distribution
- Response collection
- Review-model execution
- Difference analysis
- Final answer generation
The same interface can be integrated into agent applications and development frameworks without requiring every team to build a custom model-routing and evaluation system.
The architecture is relevant to enterprises that use a mixture of:
- Open-source models
- Proprietary models
- General-purpose models
- Domain-specific models
- Locally deployed models
- Cloud-hosted APIs
The main operational challenge is that several large models may need to remain available at the same time. That creates a much larger requirement for accelerator memory, interconnect bandwidth, scheduling, and low-latency communication.
The MetaBrain SD200 Provides the Token Engine
The MetaBrain SD200 supernode is the hardware platform positioned beneath the multi-model fusion workflow.
The system uses a multi-host, low-latency memory-semantic communication architecture. According to Inspur, it unifies and interconnects 64 domestic GPU accelerators within one system.
Its design includes:
- OCM and OAM open compute architectures
- A 3D Mesh interconnect
- Unified accelerator addressing
- High-speed peer communication
- Large shared memory capacity
- Support for multiple trillion-parameter models
- Software and hardware co-optimization
The supernode can reportedly support either a single model with up to four trillion parameters or several trillion-parameter models used together by an agent application.
Token Generation Reaches 4.77 Milliseconds in Inspur’s Test
Inspur reports that the SD200 reduced single-token generation time for the trillion-parameter Kimi K2.6 model to 4.77 milliseconds.
The company also reports a 35% reduction in time to first token compared with its earlier implementation.
These figures refer to a specific optimization and test environment. They should not be interpreted as guaranteed latency for every model, deployment, prompt length, concurrency level, or production workload.
The improvements are attributed to several techniques.
Multi-Token Prediction
Autoregressive models normally generate one token, verify the new state, and then generate the next.
Multi-token prediction attempts to generate several candidate tokens in one step and validate them together.
When prediction accuracy is high, this can reduce the number of sequential decoding rounds.
W4A8 Quantization
The optimization uses INT4 weights and INT8 activation computation for parts of the mixture-of-experts workload.
Compared with BF16 computation, this can reduce:
- Memory bandwidth pressure
- Accelerator memory use
- Compute requirements
- Per-token inference cost
Quantization may affect model quality, so production teams need to evaluate accuracy on their own workloads rather than relying only on speed results.
Just-in-Time Kernel Compilation
JIT compilation generates specialized accelerator kernels at runtime based on tensor shape, layout, and data type.
A specialized kernel can reduce unnecessary branching and improve memory access compared with a general-purpose static implementation.
Prefill-Decode Separation
Prompt prefill and token decoding have different performance characteristics.
Separating the two stages makes it possible to allocate resources differently and transfer the KV cache asynchronously, reducing contention between compute and communication.
Model Compatibility
Inspur says the SD200 has completed performance optimization for several leading open models, including:
- Kimi K2.6
- DeepSeek V4
- GLM 5.2
- MiniMax M3
Compatibility does not necessarily mean every model will achieve the same latency or throughput.
Model architecture, parameter count, MoE design, context length, quantization, batching, and serving software all influence performance.
An Enterprise Version Lowers the Deployment Threshold
A 64-accelerator supernode is suitable for major AI infrastructure projects, but it is too large and costly for many enterprises.
Inspur also introduced the MetaBrain SD200 Enterprise.
The enterprise version reduces the scale-up compute domain from 64 accelerators to 16 and is intended for local deployment of trillion-parameter models.
Its reported characteristics include:
- A 16-accelerator unified interconnect domain
- Native memory-semantic communication
- Unified addressing
- Terabyte-scale unified accelerator memory
- More than 40% improvement in time to first token for trillion-parameter inference
- Local support for leading open trillion-parameter models
- Lower migration and adaptation cost than the full SD200 system
The enterprise version is aimed at workloads such as:
- Long-document understanding
- Complex logical reasoning
- Private model deployment
- Multi-agent collaboration
- Internal research
- Sensitive business applications
It provides a smaller entry point for organizations that need local model control but cannot justify a 64-accelerator supernode.
The Infrastructure Stack Is Becoming More Integrated
The products announced at OCTS 2026 illustrate a three-layer architecture.
| Layer | Primary Responsibility |
|---|---|
| Software platform | Model access, task routing, orchestration, permissions, evaluation, and result fusion |
| CPU infrastructure | Agent processes, tool calls, sandbox execution, context management, and business-system interaction |
| GPU supernode | Large-model inference, high-throughput token generation, and multi-model execution |
The system works only when all three layers are coordinated.
A fast GPU cluster cannot compensate for weak agent scheduling. A dense CPU rack cannot improve reasoning quality without access to capable models. A sophisticated fusion API cannot deliver useful latency if the underlying models cannot be loaded or connected efficiently.
This is the major change in agent infrastructure competition.
The earlier market focused heavily on how well a server supported one large model. The agent era shifts attention toward complete-system performance:
- How many agent processes can remain online?
- How quickly can they call models and tools?
- Can several models run at the same time?
- How is memory shared or isolated?
- How are permissions enforced?
- Can the platform route simple and complex tasks differently?
- Can the system remain stable at high concurrency?
- How efficiently can it produce useful tokens?
What Enterprises Should Evaluate Before Deployment
Vendor headline numbers are useful, but they are not enough to select an agent infrastructure platform.
A production evaluation should measure the complete workload.
1. Agent Concurrency
Count not only active users but also:
- Sub-agents per task
- Tool processes
- Sandboxes
- Background jobs
- Persistent sessions
- Peak bursts
- Failure recovery processes
2. CPU and Memory per Agent
A lightweight retrieval agent and a coding agent with an isolated development environment have very different resource profiles.
The “agents per rack” figure must be mapped to the actual memory, CPU, storage, and network requirements of the target application.
3. Model Latency
Measure:
- Time to first token
- Inter-token latency
- End-to-end task duration
- Queue delay
- Tool-call delay
- Latency under concurrency
A low token-generation time in a benchmark does not guarantee a low end-to-end workflow time.
4. Multi-Model Economics
Multi-model fusion may improve quality but can multiply inference cost.
Teams should compare:
- Single-model accuracy
- Fusion accuracy
- Tokens consumed
- GPU utilization
- Response time
- Review-model cost
- Failure rate
- Human-review reduction
5. Cooling and Facility Compatibility
A native liquid-cooled rack requires suitable facility infrastructure.
Check:
- Coolant distribution units
- Supply and return temperatures
- Pressure and flow requirements
- Leak detection
- Maintenance process
- Heat reuse options
- Rack power delivery
- Data-center floor loading
6. Model and Accelerator Portability
Open module standards can improve flexibility, but actual portability depends on software compatibility.
Validate:
- Model frameworks
- Inference engines
- Quantization support
- Kernel availability
- Driver maturity
- Monitoring tools
- Accelerator replacement
- Upgrade procedures
7. Reliability and Governance
Long-running agents require strong controls around:
- Identity
- Permissions
- Secrets
- Network access
- Sandboxes
- Audit logs
- Rate limits
- Human approval
- Model routing
- Data retention
Infrastructure density should not come at the cost of operational isolation.
FAQ
What is Inspur’s CPU-native liquid-cooled agent rack?
It is a full-rack CPU server designed around the cooling system rather than adding liquid cooling to a conventional air-cooled design. Inspur says it can hold up to 384 CPUs and support more than 40,000 concurrent agents.
Does one rack really run 40,000 AI agents?
The figure is a vendor-reported maximum for Inspur’s reference architecture. Real capacity will depend on how much CPU, memory, storage, networking, and sandbox isolation each agent requires.
Why do AI agents need so many CPUs?
The language model may run on GPUs, but agents also need CPUs for scheduling, tool execution, state management, business-system access, security checks, and isolated runtime environments. Long-running and multi-agent workflows increase this CPU demand.
What is OCM 2.0?
OCM is an open compute module architecture used to separate compute modules from the wider rack design. Inspur’s liquid-cooled OCM 2.0 system supports multiple CPU architectures and full-component liquid cooling.
What is the MetaBrain SD200?
The MetaBrain SD200 is Inspur’s AI supernode for large-model inference and multi-model workloads. It uses a 64-accelerator scale-up architecture with unified addressing and high-speed interconnects.
What is the EPAI multi-model fusion API?
It is an API that sends one task to several candidate models, collects their independent answers, and uses a review-and-fusion model to identify agreement, disagreement, omissions, and unique insights before producing a final response.
Is multi-model fusion always better than one model?
No. It can improve complex tasks, but it also increases token usage, compute cost, and latency. Simple tasks are often better routed to one lightweight model.
What is the difference between SD200 and SD200 Enterprise?
The full SD200 uses a 64-accelerator scale-up domain for very large workloads. SD200 Enterprise reduces the domain to 16 accelerators and targets organizations that need local trillion-parameter model deployment at a lower infrastructure threshold.
Related Tools
- Inspur EPAI: An enterprise AI platform for model management, application development, retrieval, deployment, and multi-model workflows.
- Inspur Liquid Cooling Solutions: Infrastructure solutions for deploying liquid-cooled CPU and GPU servers.
- Open Compute Project: An industry community developing open hardware designs for servers, racks, power, cooling, and data centers.
- Kimi: Moonshot AI’s assistant and model platform, referenced in Inspur’s SD200 optimization results.
- DeepSeek: An AI model provider whose open models are included in the SD200 compatibility list.
Related Links
- Inspur Agent Infrastructure Announcement: The official release covering the liquid-cooled CPU rack, SD200, and enterprise supernode.
- MetaBrain SD200 4.77ms Performance Announcement: Inspur’s technical overview of the SD200 optimization and token-generation results.
- EPAI Multi-Model Fusion API: The official explanation of candidate generation, review, fusion, and the reported DRACO score.
- Inspur EPAI Product Page: Product information for the enterprise large-model development platform.
- Inspur Liquid Cooling Data Center Solutions: Official information on liquid-cooled data-center deployment options.
- 2026 Open Compute Technology Summit: The event site for OCTS 2026 and related open-computing announcements.
- Open Compute Project Foundation: Official information on open infrastructure standards and community projects.
Summary
Inspur’s OCTS 2026 announcements address two separate problems created by enterprise agents.
The CPU-native liquid-cooled rack focuses on scale, providing a dense environment for agent scheduling, tools, sandboxes, context management, and long-running processes. The SD200 and EPAI multi-model fusion workflow focus on intelligence, allowing several large models to work on difficult tasks while a review model combines their results.
The wider lesson is that agent infrastructure cannot be reduced to one faster accelerator. Production systems need coordinated CPUs, GPUs, memory, interconnects, cooling, orchestration, permissions, and evaluation.
The defining metric of agent infrastructure is shifting from how well one model runs to how efficiently the complete system can operate thousands of agents and produce reliable results.