Alibaba Cloud and Peking University say their Aegaeon system reduced the GPUs needed for a reported Model Studio deployment from 1,192 to 213—an approximately 82% reduction. The system also delivered up to 9x higher goodput than comparison systems in benchmark scenarios. That does not mean Alibaba made each Nvidia GPU nine times faster, or that 213 GPUs can replace 1,192 GPUs for every AI workload.
The result matters financially because it shows how better inference scheduling can reduce idle capacity and potentially lower the cost of operating large model marketplaces. Whether it reduces Nvidia’s overall sales is less certain: cheaper inference can also encourage more companies to deploy AI.
As an Amazon Associate I earn from qualifying purchases.
What Alibaba actually reported
Aegaeon is a multi-model large-language-model serving system developed by Alibaba Cloud and Peking University. According to the SOSP 2025 paper, it has been beta-deployed in Alibaba Cloud’s Model Studio marketplace for more than three months and served tens of models ranging from approximately 1.8 billion to 72 billion parameters.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →The headline claims have three separate parts:
- Resource reduction: the reported deployment requirement fell from 1,192 GPUs to 213, or roughly 82%.
- Benchmark performance: Aegaeon produced 1.5x to 9x higher goodput than ServerlessLLM and MuxServe in the paper’s experiments.
- System design: Aegaeon pools GPU capacity and makes allocation decisions at token-level granularity rather than keeping a GPU or model instance dedicated to an entire request.
Those figures should not be collapsed into the claim that “213 GPUs perform like 1,192” in every circumstance. They describe a particular model mix, traffic pattern and service objective.
#1 Best Overall
- [ Maximum AI Compute Power ] Dominate complex workloads with the ASUS ESC8000A-E13. This 4U rack server is a powerhouse engineered for mass-scale AI, machine learning, and deep training. Featuring support for dual AMD EPYC 9005/9004 processors and up to eight dual-slot GPUs, it delivers the raw computational muscle required to train LLMs and run complex simulations effortlessly. Accelerate your data science pipeline and transform raw data into actionable intelligence faster than ever.
- [ Advanced Thermal Efficiency ] High performance demands elite cooling. The ESC8000A-E13 features a cutting-edge aerodynamic design with independent CPU and GPU airflow tunnels. Equipped with redundant hot-swap fans and optimized for liquid cooling integrations, this 4U server ensures maximum uptime under heavy, sustained workloads. Keep your data center running cool, quiet, and highly efficient while preventing thermal throttling during mission-critical enterprise operations.
- [ Scale with Flexible Storage ] Future-proof your infrastructure with unmatched storage and expansion flexibility. This offers comprehensive front-panel drive bays supporting Gen5 NVMe, SAS, or SATA drives alongside multiple PCIe 5.0 slots. Designed as a high-density 4U server capable of housing eight dual-slot GPUs: NVD H200, RTX PRO 6000 Blackwell, RTX PRO 4500 Blackwell or AMD Instinct MI350P PCIe Card, each supporting up to 600 watts.
- [ Enterprise-Grade Reliability ] Minimize downtime and secure your ecosystem with server-grade redundancy. The ESC8000A-E13 is built for 24/7 continuous operation, boasting 2+2 redundant (3200W total) 80 PLUS Titanium power supplies and integrated ASUS ASMB11-iKVM for comprehensive out-of-band management. Ideal for cloud service providers, rendering farms, and large enterprise infrastructure, it combines robust physical hardware with smart remote monitoring to safeguard your digital assets.
- [Reliability Guaranteed] Shop with total peace of mind knowing that every new computer component we sell is backed by our EPC 3-year warranty. Whether you are investing in high-speed DDR5 RAM or a powerhouse GPU, we protect your build against defects and performance failures. We stand firmly behind the quality of our hardware, ensuring that your setup remains fast, stable, and secure for years to come.
Why model marketplaces waste GPU capacity
AI marketplaces often host many models, but demand is highly uneven. A small number of popular models may receive most requests, while a long tail of models is accessed only occasionally. If every model receives a dedicated GPU instance, much of that capacity sits idle between requests.
This is especially inefficient when traffic is bursty. Conventional request-level autoscaling may start or reserve an instance for a model, even though that capacity is not continuously needed. Data Center Dynamics, citing the research, reported that more than 90% of models were infrequently invoked.
Aegaeon’s central opportunity is therefore higher fleet utilization. It does not eliminate the computation needed to generate a particular answer. It attempts to share the same GPU capacity among more models and reduce idle reservation.
Recommended Free Tools
How token-level pooling works
A conventional serving system may assign a model instance or GPU to a request for the duration of that request. Aegaeon schedules more finely: each generated token becomes an opportunity to decide which model should use shared capacity next.
The architecture described in the paper has three main elements:
- Proxy layer: receives requests, distributes work and provides load balancing and fault-tolerance functions.
- Shared GPU pool: executes prefill and decoding work for multiple models.
- Memory manager: coordinates model placement, loading, switching and KV-cache-related state.
Prefill processes the user’s prompt and creates the initial attention state. Decode generates the answer one token at a time. The KV cache stores attention information needed during decoding. Sharing GPUs at this level requires careful coordination: frequent model swaps, memory movement or cache synchronization could otherwise erase the benefit.
Rank #2
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
The paper reports support for up to seven models per GPU in its evaluation. Alibaba says the technology has been applied to its Bailian/Model Studio platform, according to its company announcement.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesWhat “up to 9x higher output” really means
The more accurate term is goodput. Goodput measures useful work completed while meeting defined service-level objectives, rather than simply counting raw tokens produced without regard to latency or reliability.
In practical terms, the relevant question is: how much SLA-compliant inference traffic can a fixed GPU fleet serve? Aegaeon’s reported 1.5x-to-9x advantage is therefore an effective-capacity comparison against particular baselines—not proof that every request generates tokens nine times faster.
The paper also reports 2x-to-2.5x higher request-arrival rates under comparable conditions. These are valuable results for a multi-model marketplace, but they remain workload- and configuration-dependent.
Separating the deployment result from the test hardware
| Context | Reported details |
|---|---|
| Production beta deployment | Model Studio; tens of models; approximately 1.8B to 72B parameters; 1,192 GPUs reduced to 213 |
| Paper evaluation | Compared with ServerlessLLM and MuxServe; up to seven models per GPU; 1.5x–9x higher goodput |
| Reported production GPU | Nvidia H20 GPUs, according to coverage of the deployment |
| Separate experimental testbed | Two nodes with 16 Nvidia H800 80GB GPUs, NVLink, 2TB DDR5 memory and 192 Intel Xeon Platinum 8469C CPUs |
The 1,192-to-213 deployment figure should not be merged with the separate 16-GPU H800 testbed. They describe different contexts.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Where Aegaeon could produce the largest savings
The system is most likely to help when a provider has a large catalog of models, unpredictable or bursty demand, many lightly used models and centralized control over model storage, memory and scheduling.
Rank #3
- AI-Optimized: Designed to support up to 4 GPUs, it is perfect for handling intensive AI and machine learning tasks, ensuring high performance and scalability for advanced computational needs.
- Intelligent Storage: Equipped with 8 hot-swappable 3.5" SATA/SAS drives (12Gbps), featuring SGPIO and temperature control, it ensures efficient data management and reliable storage performance.
- Robust Cooling: The system includes 3x 12038 hot-swap PWM fans and 2x 8038 rear fans, providing advanced thermal management to maintain optimal temperatures and ensure stable operation under heavy workloads.
- Rack-Ready: Comes with a pre-installed rail kit, allowing for quick and easy installation in standard 19-inch server racks, making it ideal for data center environments and enterprise setups.
- Versatile Connectivity: Offers USB 3.0 and the latest USB 3.2 Type-C ports, ensuring high-speed data transfer and compatibility with a wide range of peripherals and devices for enhanced connectivity options.
- Multi-tenant model marketplaces
- Serverless LLM endpoints
- Cloud fleets with substantial long-tail traffic
- Providers trying to serve more models without adding GPUs linearly
- Workloads where aggregate SLA-compliant throughput matters more than the lowest possible latency for every request
The likely economic benefits include fewer idle instances, better capital utilization and potentially lower power and cooling demand per unit of served inference. However, the available evidence does not establish an 82% reduction in electricity costs, total cost of ownership or customer pricing.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Important limitations
It may not help a continuously busy model
If one model is popular enough to keep GPUs fully occupied, pooling provides less idle capacity to recover. The biggest gains are likely to come from uneven demand across many models, not from a single model running continuously at peak utilization.
Model switching can be expensive
Weights may need to move between GPU memory, host memory and storage. Large models and large KV caches increase memory pressure. Cold starts, eviction and synchronization can raise latency if the system cannot keep frequently used state close to the GPUs.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Higher goodput does not guarantee lower tail latency
A scheduler may increase aggregate throughput while still causing some requests to experience worse p95 or p99 latency during contention, model loading or queueing. Different models may also have different time-to-first-token and output-generation targets.
It is not a training breakthrough
Aegaeon addresses inference serving. It does not reduce the GPU requirement for pretraining or fine-tuning a model, and it does not make an individual GPU intrinsically more powerful.
Reproduction is not automatic
The result may depend on Alibaba’s integrated model repository, storage hierarchy, networking and traffic patterns. Tom’s Hardware noted that the available paper and reporting do not fully specify every network-fabric detail needed to reproduce the deployment. Independent replication would require the model mix, traffic traces, memory configuration, load timings, latency distributions and deployment settings.
Rank #4
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
How it compares with other approaches
Aegaeon is not the only way to improve GPU utilization.
Free tools Windows power users keep installed
One-click scans. No signup required.
- ServerlessLLM: focuses on efficient model loading and serverless serving; it is one of Aegaeon’s paper baselines.
- MuxServe: represents multi-model multiplexing and co-location approaches, also used as a comparison baseline.
- Nvidia Run:ai: provides GPU orchestration and fractional allocation. Nvidia and Nebius reported up to three times more users in a separate mixed-LLM benchmark, but that result is not directly comparable with Aegaeon’s 9x goodput figure. See Nvidia’s technical post.
- Nvidia Dynamo: is a broader inference-orchestration platform for large AI factories and may complement specialized serving systems rather than replace them. Nvidia describes it at Nvidia News.
- MIG and time-slicing: can divide or share GPUs, but generally operate at a coarser level than token-based model scheduling.
What this means for Nvidia
The immediate interpretation is mixed. Better pooling could reduce the number of GPUs needed to serve a given catalog of models, potentially delaying purchases by cloud providers. That would be a negative for unit demand if the workload stayed fixed.
But lower inference costs can also make AI applications affordable for more customers. More applications, longer outputs and broader model availability could increase total demand for compute. Aegaeon therefore challenges inefficient GPU utilization—not necessarily Nvidia’s entire accelerator market.
It is also important that the reported system still uses Nvidia hardware, including H20 GPUs in the production deployment described by coverage. The technology is a serving and scheduling optimization, not a replacement accelerator.
What companies can actually buy
Organizations evaluating this area should distinguish between several product categories:
- Managed model platforms: hosted marketplaces such as Alibaba Cloud Model Studio/Bailian.
- GPU orchestration: tools such as Nvidia Run:ai for allocation and fleet management.
- Inference orchestration: platforms such as Nvidia Dynamo for coordinating large-scale serving infrastructure.
- Open-source serving stacks: systems operated directly by a company on its own GPU cloud or data center.
- Dedicated GPU endpoints: simpler deployments that may be preferable when one model has sustained demand or strict isolation requirements.
No listed product should be assumed to reproduce Aegaeon’s 82% reduction. The relevant comparison depends on model sizes, traffic distribution, latency targets, storage speed, memory capacity and whether customers require dedicated placement.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
