Recommended Free Tools
AI inference does not have one fixed price per request. A hosted API may charge separately for input and output tokens, while self-hosted inference depends on how much capacity you keep available and how much work that capacity actually serves. To compare the two, match the model, request lengths, service level and latency target—and include the full cost of keeping a self-hosted system running.
What drives the cost of serving each request?
The main drivers are the model and hardware, the amount of input and output, the context retained during generation, batching and utilization, and the latency and availability the service must provide. These factors interact: two requests with the same total token count can require different resources, and a GPU’s hourly price alone does not reveal the cost of the useful output it delivers.
As an Amazon Associate I earn from qualifying purchases.
Model and serving hardware
A larger or more demanding model may need more accelerator memory or compute. The relevant comparison is achieved throughput and output quality on the target workload and latency requirement—not simply the hourly price of a GPU. NVIDIA frames inference cost as an end-to-end measure that includes GPUs, CPUs, networking, software and ecosystem. That is useful vendor positioning, not an independent rule proving which product is cheapest. NVIDIA puts it this way: Only looking at compute pricing or FLOPs per dollar gives an incomplete view of inference TCO.
NVIDIA’s AI inference materials.
Input and output length
Hosted services often charge different rates for input and output tokens. A long prompt takes work to process and expands the context the system must retain; a long answer extends generation work. Therefore, the same total token count can have different resource demands depending on how those tokens are split between prompt and response.
#1 Best Overall
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
Context and KV cache
During generation, a model uses a key-value (KV) cache to retain attention state. Microsoft Research’s Splitwise paper notes that each active generated token accesses the cache for the context so far. In the paper’s studied systems, prompt batching was compute-bound while token generation was limited by memory capacity. This helps explain why long contexts and many concurrent sequences can make memory capacity and bandwidth important to throughput; it is a finding about the paper’s models and systems, not a universal performance guarantee. Microsoft Research’s Splitwise paper.
Batching, concurrency and utilization
Serving several requests together can spread fixed work across more tokens and improve hardware efficiency. But batching is constrained by latency targets, context sizes, memory and the mix of requests arriving. A system that must respond quickly may have less room to wait for a large batch; a system serving light or bursty demand may keep expensive reserved capacity idle.
A 2026 study of H100 and H200 systems illustrates why a per-token energy result must be read with its workload details. For Llama-3.2-1B on H200 at batch 16 and context 4K, the authors report that increasing output length from 10 to 512 tokens reduced token energy from 7.46 to 0.72 joules per token, while total energy in the batched inference window rose from 1.19 to 5.93 kJ. In that experiment, longer output spread energy across more generated tokens even as the window used more energy. It is not a general prediction that longer requests cost less. The 2026 study.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Latency, availability and power
Low-latency service can require capacity to be ready before requests arrive. Work that can be scheduled flexibly may use capacity more efficiently, but the result depends on the workload and provider. Electricity also matters to data-center operators, yet there is no useful generic electricity cost per AI request without specifying hardware, power draw, utilization, facility overhead and electricity price. Microsoft Research notes that peak power draw directly affects data-center cost in its systems discussion; the 2026 study measures energy under specific combinations of model, phase, batch size, context and output length.
Rank #2
- Powered by Radeon AI PRO R9700 - Supercharge you workflow with the cutting-edge RDNA 4 Architecture and 2nd-gen AI Accelerators.
- 32GB GDDR6 with 256-bit memory bus - Tackle larger, more complex projects without limits.
- PCIe Gen 5 - Unlock lightning-fast data transfers with PCIe Gen 5 support.
- GIGABYTE TURBO Fan Cooling System - Indented metal cover and blower fan increase airflow intake, while the vapor chamber, all copper heat sink, and metal frame offer efficient heat dissipation. Optimized airflow design allows for easy multi-GPU scalability.
- Double Ball Bearing Fan - Delivers superior heat resistance and rotational efficiency for better performance and a longer lifespan compared to conventional sleeve fans.
How much does AI inference cost per request?
For a usage-metered hosted API, a useful starting estimate is:
Request charge ≈ input tokens × input rate + output tokens × output rate + applicable cache, tool or service-tier charges.
It is only a starting point. Providers may meter cache reads and writes, long-context requests, batch or fast modes, and image or audio inputs differently. Check the provider’s current rate table for the exact model and service tier rather than assuming that a quoted token rate covers every feature.
For a dated example of how meters differ, DigitalOcean’s machine-learning pricing page lists model-specific rates per million tokens and GPU-hour prices for dedicated inference. It also states that batch inference can be discounted by up to 50% for OpenAI and Anthropic models. That maximum is a provider-specific offer, not a standard market discount or a guarantee that every request qualifies. Prices and availability can change; consult the page for current terms.
Rank #3
- [Local AI Inference & 70B Model Ready] Equipped with the AMD Ryzen 7 PRO 8845HS processor, NEXUS is engineered for heavy local AI workloads. With a full-size GPU bay, it runs 70B LLMs natively without an internet connection. Ideal for AI developers and tech enthusiasts who need private environment for coding and model testing.
- [132TB Mass Storage with ZFS Integrity] Features a hybrid storage architecture (3×NVMe + 4×3.5" HDD) supporting up to 132TB. Utilizing the enterprise-grade ZFS file system and ECC memory, it prevents data corruption and bit rot—a must-have for professional photographers and video editors safeguarding 4K/8K RAW footage.
- [OpenClaw-Driven Automation Workflow] The built-in OpenClaw execution layer allows complex automated tasks to be processed locally. Even when offline, your backup schedules and AI file organization continue seamlessly. Say goodbye to monthly cloud subscriptions and high latency.
- [Dual 10GbE & USB4 Ultra-Connectivity] Experience server-class speeds with dual 10GbE ports and a 40Gbps USB4 interface. It enables multi-user real-time collaboration on large project files directly from the NAS, ensuring zero-lag editing for creative studios and production teams.
- [Open-Source ZimaOS for Total Privacy] Running on the fully open-source ZimaOS, NEXUS ensures your data stays physically on-premise with no backdoors. It acts as a "Digital Fortress" for privacy-conscious families and small businesses who demand absolute data sovereignty.
A dedicated GPU-hour price cannot be compared directly with a per-token API rate. You need to know how many tokens the system actually serves during the billed time, for the target model and request pattern, and account for capacity that remains reserved between requests.
Why are input and output tokens priced differently?
Input tokens are processed as the model reads the prompt and builds its context; output tokens are generated sequentially, with each new token drawing on the context accumulated so far. Those phases stress the system differently. The KV-cache behavior described in Splitwise helps explain why generation can put substantial pressure on memory, while prompt processing can be more compute-intensive in the paper’s setup.
Providers set their own rates and meters, so the technical difference does not imply one universal input-to-output price ratio. Use the specific model’s published rates, and estimate the two token counts separately. Include any cache, context-length or service-mode pricing that applies.
How does GPU utilization affect cost per token?
Utilization determines how much served work shares the cost of capacity. Two measures clarify the difference:
Rank #4
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
- Usage-based cost: infrastructure cost attributed to active inference work divided by the tokens processed.
- Allocation-based cost: the cost of all infrastructure needed to keep the model available—including reserved GPU capacity and shared services—divided by the work actually served.
CNCF’s OpenCost discussion illustrates the gap with an example: $1.00 usage-based cost versus $4.00 allocation-based cost per million tokens implies 25% utilization in that example. These are explanatory figures, not an industry benchmark. For a build-versus-buy decision, the allocation-based view is the relevant one because the reserved capacity still has a cost when it is idle. CNCF notes: Both metrics can also be expressed as cost per million tokens, but they answer different questions and should never be confused.
CNCF’s article about OpenCost inference cost tracking.
Is self-hosted inference cheaper than an API?
It can be, but a comparison based only on GPU compute or an API’s token rate is incomplete. Self-hosting requires allocating the full cost of the system across the work it serves. Depending on the deployment, that can include rented or amortized hardware, reserved capacity, shared services, networking, storage, operations and resilience. The calculation should reflect real traffic, including idle periods, not just peak throughput during active inference.
Compare the options on the same basis:
| Approach | Typical cost basis | Questions to answer |
|---|---|---|
| Managed inference API | Often input and output token rates, with possible cache, tier or batch distinctions | Which model and rate tier? What are the input and output token counts? Do tools or service features add charges? |
| Dedicated cloud inference | GPU-hour or instance time, plus platform costs | How much capacity must remain available? What utilization, latency and scaling behavior are realistic? |
| Self-hosted infrastructure | Amortized or rented hardware plus operating costs and idle or allocated capacity | What is fully allocated cost per token at observed traffic? Which staffing, network, storage and resilience costs belong in the calculation? |
Before choosing, make the comparison like for like:
- Fix the service target. Use the same model quality, request pattern, output length and latency requirement for each option.
- Estimate API charges from actual usage. Separate input and output tokens, then apply the current model rates and any relevant cache, tool, tier or batch charges.
- Measure self-hosted throughput under that workload. Include the context lengths, concurrency and response-time target that matter in production.
- Allocate all capacity and operating costs. Divide by the tokens actually served over the period, including the effect of idle or reserved capacity.
- Compare equivalent outcomes. A cheaper configuration is not equivalent if it fails the required quality, latency or availability target.
Vendor benchmark figures need the same care. Google Cloud’s 2023 blog reported 1.7×–3.9× relative performance improvement for specified H100/A3 workloads over A2 and up to 1.8× performance per dollar for a specified L4 comparison. Google explicitly said its derived performance-per-dollar measure was not an official MLPerf metric and had not been verified by MLCommons. Those historical, benchmark-scoped results are useful for understanding methodology, not current purchasing guidance. Google Cloud’s benchmark discussion.
What to include in an inference cost estimate
- Model, hardware and software configuration, measured at the quality and latency you need.
- Input tokens, output tokens, context length and concurrency—not just a single average token count.
- Provider-specific rates, including any cache, long-context, tool, modality or service-tier charges.
- Batching and utilization assumptions, checked against actual traffic patterns.
- For self-hosting, reserved capacity and shared infrastructure as well as active compute.
- For energy estimates, the hardware, power draw, facility overhead and electricity price behind the figure.
There is no single supported figure for the electricity cost of an AI request, nor a universal cost per request that applies across models and workloads. A useful estimate is specific to the service, request shape, pricing terms and period being evaluated.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
