How to Choose a Cloud GPU Instance for AI Workloads in 2026

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

The right cloud GPU is the least expensive instance that can hold your model and deliver the required throughput or latency reliably. Start with the workload, set a GPU-memory floor, decide whether you need one GPU or a distributed cluster, then compare measured cost per completed step, image, request, or million tokens. An H100 is not automatically a better purchase than an A100, and a GPU is not automatically cheaper than a CPU.

For a sound decision, price the whole run: accelerator time, vCPUs and system RAM, storage, data transfer, software licenses, idle periods, checkpointing, and the risk of Spot interruption. Provider availability, quotas, drivers, interconnects and regional capacity can matter as much as the advertised hourly rate.

As an Amazon Associate I earn from qualifying purchases.

What are you actually running on the GPU?

GPU selection follows the job. A pre-training cluster has very different requirements from a small real-time model or a graphics workstation. Write down the objective, deadline, acceptable interruption rate and output unit you will use for cost accounting before looking at instance names.

Pre-training and large distributed training

Pre-training usually needs several GPUs, fast GPU-to-GPU links, high-bandwidth networking and enough host memory to feed the accelerators. Start by evaluating H100-, H200- or B200-class multi-GPU families when the model, parallelism strategy and budget justify them. Verify scaling on your exact framework; eight GPUs do not guarantee eight times the throughput.

Fine-tuning and medium-size models

Fine-tuning often fits on one A100, H100 or an equivalent accelerator, but memory is the first constraint. Quantization, gradient accumulation and parameter-efficient methods can reduce the working set, yet you should leave room for framework overhead and a larger batch or sequence length later.

Batch inference

Batch jobs can trade latency for utilization. L4, A100, H100 and comparable instances may all be economical depending on model size and batching efficiency. Spot or preemptible capacity is attractive when the job can checkpoint and restart without losing expensive progress.

Real-time serving

For an API, measure p95 latency, concurrency, throughput and cost per request rather than raw FLOPS. Google Cloud notes that quantized formats such as FP8, INT8 and INT4 can maximize request throughput per dollar. Test L4/G2, G4, AWS G5/G6, or Azure NC/ND options with the quantization and serving stack you intend to deploy.

Graphics, visualization and HPC

Graphics and virtual-desktop workloads often favor graphics-oriented families, while HPC may depend on double-precision performance, memory bandwidth and a specific interconnect. Azure positions NV families for visualization and virtual desktops, and NC families for compute-intensive AI/ML and HPC.

CPU-friendly pipelines

Small models, feature engineering, preprocessing and some classical ML pipelines can be faster or cheaper on CPU instances. Benchmark a representative workload; “GPU” is not a synonym for lower cost.

How much GPU memory do you need?

Set a memory floor before comparing accelerator generations. The GPU must hold the complete working set, not just the model file.

Build the working-set estimate

• Weights: include the number of parameters and the precision used at runtime or during training.
• Activations: account for batch size, sequence or image dimensions, and whether activation checkpointing is enabled.
• Optimizer state: full-precision optimizers can require several times the weight memory during training.
• Framework overhead: reserve space for kernels, communications buffers, CUDA graphs and runtime allocations.
• Growth margin: leave capacity for a larger batch, longer context, new layers and checkpoint loading.

A model that exceeds available memory will fail regardless of theoretical compute. If the estimate is close to the card limit, choose the next memory tier or redesign the workload before committing to a long run.

Current accelerator memory figures

Accelerator	Memory	Cloud mapping in the cited 2026 documentation
NVIDIA L4	24 GB GDDR6	Google Cloud
NVIDIA A100	40 GB HBM2	Google Cloud A2 Standard
NVIDIA A100	80 GB HBM2e	Google Cloud A2 Ultra
NVIDIA H100	80 GB HBM3	Google Cloud A3 families
NVIDIA H200	141 GB HBM3e	Google Cloud A3 Ultra
NVIDIA B200	180 GB HBM3e	Google Cloud A4

These are accelerator capacities, not the RAM available to your operating system or data loader. Check the instance specification for host CPU, system memory, local disks and attached-storage bandwidth separately.

Should you use one GPU or a multi-GPU instance?

Use one GPU when the model and deadline fit comfortably on a single card. Move to four- or eight-GPU instances only when parallelism reduces completion time enough to justify the extra accelerator, host and networking cost.

Single-GPU workloads

One-GPU instances simplify deployment, debugging and quota requests. They are a practical default for inference, many fine-tuning runs and experiments where utilization is uncertain. Fractional-GPU options can lower the bill for light inference, but confirm that the fraction supplies enough memory and predictable latency.

Distributed training

Distributed jobs need more than a count of GPUs. Check NVLink or an equivalent high-bandwidth interconnect, network bandwidth, collective-communication support, host RAM and storage throughput. AWS documents eight-GPU configurations in P4d, P4de and many P5 variants; Google Cloud documents eight-GPU configurations across A2 and A3 families. Treat those as instance layouts, not a promise of linear scaling.

Parallelism and failure domains

Data, tensor and pipeline parallelism place different demands on memory and communication. A topology with GPUs on separate hosts may incur network latency that erases the benefit of adding cards. Confirm how the provider places GPUs and whether your scheduler can restart an entire worker group after a host failure.

What do the major cloud GPU families offer?

Provider names change, but the family roles below are documented in 2026 guidance. Availability and quotas vary by region, so verify the exact SKU before budgeting.

Provider	Families and documented accelerators	Typical positioning
AWS	G4, G5, G6 and G7; Inf1 and Inf2; P4d/P4de; P5/P5e/P5en; newer P6 variants	Graphics and inference; AWS inference accelerators; A100 and H100/H200 training configurations
Azure	ND, NC and NV families	ND for deep learning and large-memory generative-AI training; NC for compute-intensive AI/ML and HPC; NV for visualization, virtual desktops and some foundational AI
Google Cloud	A4, A3 Ultra, A3 Mega/High/Edge, A2 Ultra, A2 Standard, G4 and G2	B200, H200, H100, A100 80 GB, A100 40 GB, RTX PRO 6000 and L4 mappings respectively

Microsoft Azure describes the reason for these specialized families directly: “AI workloads require specialized virtual machines (VMs) to handle high computational demands and large-scale data processing.” Small models may still be better on CPU instances, even when a GPU family is available.

Is an H100 worth it over an A100?

Sometimes, but only when the H100’s measured throughput or shorter completion time offsets its higher instance cost. Memory capacity, software kernels, precision support, interconnect and your utilization matter more than the generation label alone.

Choose an A100 when

• Your working set fits in 40 GB or 80 GB and the A100 instance is materially cheaper or easier to obtain.
• Your framework, kernels or licensed software are already validated on A100.
• The workload is throughput-insensitive, lightly utilized or frequently paused.

Choose an H100 or newer accelerator when

• Faster completion reduces a deadline, queue or staff cost enough to outweigh the rate premium.
• Your model benefits from newer precision modes and optimized kernels.
• You need more effective interconnect bandwidth for distributed training or higher serving concurrency.

Run an identical representative workload on both when possible. Compare completed training steps per dollar, images per dollar, requests per dollar or million tokens per dollar; a peak-FLOPS comparison is not a financial decision.

What is the cheapest GPU for inference?

There is no universal cheapest card. The least-cost option is the one that meets your memory and p95-latency target at your actual concurrency, including host and idle costs.

Test an efficient serving tier first

L4/G2 and graphics-oriented G4 or AWS G5/G6 families can be strong starting points for smaller, quantized models. Azure NC or ND may be preferable when the model needs more memory or a particular software stack. Measure warm-up time, batching behavior, p95 and p99 latency, error rate and cost per request.

Account for utilization

An inexpensive hourly GPU that serves a few requests per minute can cost more than a smaller card or CPU instance. Include autoscaling floor capacity, replicas, health checks, logging, model-download time and periods with no traffic.

Use quantization deliberately

FP8, INT8 and INT4 can increase throughput and reduce memory, but quality and kernel support vary by model. Validate accuracy and tail latency on production-like inputs before treating the lower memory requirement as a saving.

How should you compare the total cost?

Compare the cost of a completed outcome, not the posted GPU-hour. A useful calculation is:

Total run cost = accelerator time + vCPU/RAM time + storage + data transfer + licenses + checkpoint/restart overhead + idle capacity.

Choose the right denominator

• Training: cost per completed optimizer step or per successful run.
• Image or video generation: cost per accepted output at the required quality.
• Batch inference: cost per million tokens, records or images.
• Online serving: cost per request at the target p95 latency and concurrency.

Build a run worksheet

Line item	What to record	Why it changes the decision
GPU	Instance rate, GPU count, utilization and actual run hours	Idle or underused cards dominate many small workloads
CPU and RAM	vCPUs, system memory and host-hour rate	Data loading and preprocessing can bottleneck the GPU
Storage	Boot disk, datasets, checkpoints, I/O rate and retention period	Slow storage extends billed runtime; retained checkpoints add recurring cost
Network	Ingress, egress and inter-zone or inter-region traffic	Moving large datasets or serving across regions can exceed compute cost
Reliability	Checkpoint frequency, restart time and expected interruptions	Lost work raises the effective cost of discounted capacity
Software	Licenses, support plans, managed-service fees and images	A cheaper card may require a more expensive software path

Run at least one production-like pilot and record throughput, memory use, p95 latency and utilization. Provider prices, regional capacity, quotas and marketplace listings change, and no provider-neutral price-per-token benchmark is universal.

When do Spot or preemptible GPUs make sense?

Use discounted capacity for restartable work. Google Cloud documents Spot discounts of 60–91% below on-demand rates, but Spot VMs can be stopped or deleted with 30 seconds’ notice.

Good candidates

• Batch inference with independent shards.
• Training that checkpoints frequently to durable storage.
• Hyperparameter searches where failed trials can be relaunched.

Risk controls

1. Write checkpoints to storage that survives VM deletion.
2. Test restoring a checkpoint on a fresh instance.
3. Use an automatic termination handler and a queue that resubmits failed work.
4. Keep an on-demand fallback for deadlines or scarce accelerator types.
5. Include restart time and partially completed work in cost-per-result calculations.

A nominal 90% discount is not a 90% saving if interruptions repeatedly discard hours of computation or leave an expensive replica running while a replacement is found.

What operational checks should happen before purchase?

Confirm the following before reserving capacity or moving data:

• Region: the exact GPU SKU, attached storage and networking features exist where your data and users are located.
• Quota: your account has enough accelerator quota for the requested count; quota increases may require lead time.
• Image and driver: the operating-system image, CUDA or other runtime, framework and driver versions are compatible.
• Topology: GPUs share the expected interconnect and hosts have enough network bandwidth.
• Storage bandwidth: the dataset and checkpoint path can feed every GPU without stalls.
• Reservation: committed or reserved capacity matches the period you can actually use it.
• Interruption policy: Spot or preemptible behavior, notice period and restart semantics are understood.
• Data movement: egress, encryption, access controls and retention meet your requirements.

How do you choose an instance step by step?

1. Define the output. Record whether you need training steps, completed experiments, tokens, images or requests, plus deadline and latency targets.
2. Measure the working set. Estimate weights, activations, optimizer state, batch size, framework overhead and growth margin. Reject every GPU below the memory floor.
3. Select a scale. Choose one GPU or a fractional option for a small workload; choose four to eight GPUs only when parallelism and deadline justify it.
4. Shortlist families. Map the required memory, precision and software support to the provider families and regions available to your account.
5. Benchmark. Run the same data and code on two or more candidates. Capture throughput, p95 latency, utilization, host bottlenecks and failure behavior.
6. Calculate total cost. Add GPU, vCPU/RAM, storage, transfer, licenses, idle time and expected restart overhead, then divide by the completed output.
7. Test operations. Verify quota, image, drivers, storage, topology, checkpoint restoration and autoscaling before a long run.
8. Choose purchasing mode. Use on-demand for uncertain experiments, reservations for predictable sustained use, and Spot or preemptible capacity for restartable jobs with a fallback.
9. Recheck at deployment. Confirm current regional availability, price and quota immediately before launching; provider catalogs change.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Which instance strategy fits common workloads?

Workload	Starting strategy	Measure before committing
Large distributed training	H100, H200 or B200-class multi-GPU family	Scaling efficiency, interconnect bandwidth, host RAM and checkpoint recovery
Fine-tuning or medium model	A100 40/80 GB, H100 or equivalent; select memory first	Peak memory, tokens per second and cost per completed step
Real-time inference	L4/G2, G4, AWS G5/G6 or Azure NC/ND with tested quantization	p95 latency, concurrency, accuracy and cost per request
Batch processing	Spot or preemptible GPU with frequent checkpoints	Interruption rate, restart time and cost per completed item
Small or CPU-friendly model	CPU baseline before adding a GPU	End-to-end throughput and monthly cost at expected utilization

What common mistakes increase the bill?

Buying compute before checking memory

A high-FLOPS card that cannot load the model wastes setup time and may fail during the first large batch. Calculate the working set and reserve headroom first.

Comparing hourly rates only

Hourly price ignores utilization, storage, data transfer, host resources and lost work. Use a completed-output denominator.

Assuming multi-GPU scaling is linear

Communication, synchronization and input pipelines can leave additional GPUs idle. Benchmark the exact parallelism strategy and topology.

Ignoring quota and regional scarcity

A theoretically ideal SKU is not useful if your account cannot obtain it in the required region or timeframe. Keep a technically compatible fallback.

Running Spot without recovery tests

Checkpoints that have never been restored are not a resilience plan. Simulate termination before relying on discounted capacity.

Leaving development instances running

Set automatic shutdowns, budget alerts and labels for owner, project and expiration. Review utilization and delete unattached disks, snapshots and idle replicas.

What is the practical verdict?

Choose the smallest current GPU that clears your memory floor, meets measured throughput or p95 latency, and is available with an acceptable interruption and quota risk. For distributed training, pay for H100-, H200- or B200-class capability only after validating scaling; for fine-tuning, an A100 40/80 GB may be the better financial choice when it fits; for inference, start with L4/G2, G4, AWS G5/G6 or Azure NC/ND candidates and benchmark quantized serving; for batch work, use Spot or preemptible capacity with tested checkpoints. Recalculate total cost whenever provider prices, regional supply, software versions or workload traffic changes.

Frequently Asked Questions

Can I move the same model between cloud GPU providers?

Usually, model files and high-level framework code are portable, but container images, drivers, kernel optimizations, networking and managed-service APIs may not be. Keep an exportable environment and test a clean deployment on the fallback provider before an outage makes portability urgent.

How often should I rerun a GPU comparison?

Repeat the benchmark when you change model size, precision, batch or context length, serving engine, framework version or region. A quarterly review is useful for steady production workloads because accelerator availability and pricing can change sooner than your code does.

Can different GPU models share one distributed training job?

Mixed generations can create synchronization and memory-capacity problems, and many distributed-training configurations assume identical workers. Use homogeneous workers unless your framework explicitly supports heterogeneity and you have measured the resulting efficiency.

What should a small team monitor after deployment?

Track GPU utilization, memory headroom, queue time, p95 latency, completed outputs, restart count and spend by project. Alert on sustained idle capacity and on costs per request or training step moving above the benchmark baseline.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.