Recommended Free Tools
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
If a GPU cloud provider fails, running jobs can stop immediately and new GPU capacity may be unavailable. Whether you experience a delay, lose recent training progress, or lose data depends on the failure scope and where your files, checkpoints, and environment are stored. Data on instance-local temporary storage can disappear during host maintenance or a restart, while separately stored and verified copies may remain intact. Service-level agreement (SLA) credits can provide a contractual remedy, but they do not restore compute, recover deleted data, or keep your business operating. The practical defense is to separate compute from durable storage, set explicit recovery time and recovery point objectives, verify restores, rehearse failover, and confirm where replacement GPU capacity will come from.
What can fail when a GPU cloud provider goes down?
“The provider is down” can describe several different incidents. A failed GPU card, a host-maintenance restart, a zone outage, a storage or network interruption, a control-plane problem, an account suspension, and a provider-wide business failure have different consequences and recovery paths.
As an Amazon Associate I earn from qualifying purchases.
| Failure scope | Likely effect on workloads | Primary question to answer |
|---|---|---|
| GPU hardware or host fault | A training or inference process can stop; the virtual machine may need a reset or reboot. | Is a usable checkpoint available outside the affected VM? |
| Host maintenance or restart | The instance stops. Temporary local-disk data can be lost even if the VM returns. | Which files were stored only on instance-local storage? |
| Zone, region, network or storage outage | Compute may exist but be unreachable, or jobs may be unable to read data and checkpoints. | Can the workload run in another location with the same data access? |
| Provider control-plane incident | You may be unable to create, start, stop or monitor instances even when hardware is operating. | Do you have an independent way to reach logs, credentials and support? |
| Account suspension or quota failure | Capacity can be denied for administrative rather than technical reasons. | Who can resolve the account issue, and is a second account or provider available? |
| Provider business failure | Access, support and data retrieval may be threatened at the same time. | Can you export all important data and rebuild elsewhere without provider-specific dependencies? |
Compute availability and data durability must therefore be assessed separately. A stopped VM does not automatically mean a lost dataset, and an intact storage service does not guarantee that a running process can resume.
Will you lose training data and checkpoints?
Not necessarily. You lose data when the only copy is on storage that the incident destroys or makes unrecoverable. Durable copies in a separate storage service, region or provider can survive a compute failure, provided the copies are complete, accessible and restorable.
#1 Best Overall
- Get NVMe solid state performance with up to 1050MB/s read and 1000MB/s write speeds in a portable, high-capacity drive(1) (Based on internal testing; performance may be lower depending on host device & other factors. 1MB=1,000,000 bytes.)
- Up to 3-meter drop protection and IP65 water and dust resistance mean this tough drive can take a beating(3) (Previously rated for 2-meter drop protection and IP55 rating. Now qualified for the higher, stated specs.)
- Use the handy carabiner loop to secure it to your belt loop or backpack for extra peace of mind.
- Help keep private content private with the included password protection featuring 256‐bit AES hardware encryption.(3)
- Easily manage files and automatically free up space with the SanDisk Memory Zone app.(5). Non-Operating Temperature -20°C to 85°C
Why instance-local storage is risky
Google Cloud states that GPU instances stop during host maintenance. When Local SSD is attached, its data is lost after the stop; Google says it cannot recover that data after a host-maintenance restart and advises against using Local SSD for data with strong persistence requirements. Its guidance is specific to Google’s Local SSD product, but the design lesson applies broadly: treat ephemeral, instance-local disks as scratch space, not as the only home for datasets, model artifacts, logs or checkpoints.
Google’s wording is direct: “Avoid storing data with strong persistency requirements on Local SSD.” A checkpoint written only to that disk is not a backup. A checkpoint copied to a separately managed persistent store, downloaded to a controlled local system, or replicated to another provider is a recoverable asset if you have tested the copy.
What should be copied?
- Source code and exact dependency or container definitions.
- Training and inference datasets, including manifests and preprocessing code.
- Model checkpoints, optimizer state and configuration files.
- Secrets, credentials and identity policies stored through a secure, recoverable process.
- Job specifications, infrastructure definitions, launch scripts and scheduler settings.
- Logs and metrics needed to identify the last successful checkpoint and reproduce a run.
Copying only the final model is insufficient if rebuilding requires a particular library version, tokenizer, dataset revision or network configuration. Record those dependencies with the artifact they produced.
What does an SLA actually promise after an outage?
An SLA defines a provider’s measured service commitment and the remedy available when that commitment is missed. It does not universally promise that your job will resume, that data will be restored, or that equivalent GPUs will be available during an incident.
Rank #2
- MADE FOR THE MAKERS: Create; Explore; Store; The T7 Portable SSD delivers fast speeds and durable features to back up any endeavor; Build your video editing empire, file your photographs or back up your blogs all in an instant
- SHARE IDEAS IN A FLASH: Don’t waste a second waiting and spend more time doing; The T7 is embedded with PCIe NVMe technology that brings fast read and write speeds up to 1,050/1,000 MB/s¹, making it almost twice as fast as the T5
- ALWAYS MAKE THE SAVE: Compact design with massive capacity; With capacities up to 4TB, save exactly what you need to your drive – from large working files to game data and everything in between
- ADAPTS TO EVERY NEED: Whether using a PC or mobile phone, count on the T7 for extensive compatibility²; It’s a true team player when it comes to heavy-duty application usage or file-saving
- HI RESOLUTION VIDEO RECORDING: Record Ultra High Resolution (4K 60fs) videos directly onto the T7 Portable SSD with your favorite camera or mobile devices; Supports iPhone 15 Pro Res 4K at 60fps video and more³
NVIDIA’s terms are offering-specific
NVIDIA’s Cloud Services SLA, last modified November 5, 2025, calculates availability over a calendar month and sets details by offering. It lists a 99% monthly service-availability target for NVIDIA Omniverse Cloud; that figure is not a general promise for every GPU-compute service.
For a critical issue, NVIDIA says it will use commercially reasonable efforts to find a resolution within 24 hours after determining that the incident is critical. The SLA expressly says this is a target that may not always be achieved, so it is not a guaranteed workload-recovery time. General terms exclude causes such as force majeure and matters outside NVIDIA’s reasonable control. A customer claim must include incident details and reach NVIDIA within two months of the incident’s start. The applicable offering determines the remedy and any availability target.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →A component guarantee may exclude the GPUs
KDDI GPU Cloud’s published SLA illustrates why you must read the component table rather than rely on a headline percentage. Its terms list 99.95% monthly availability for high-speed storage and the service network, while marking GPU servers outside that availability guarantee. A separate hardware-replacement commitment applies under stated conditions. This is a KDDI-specific example, not an industry-wide GPU-server standard.
What to record before signing
- The exact service and components covered: GPU servers, storage, network, control plane and support.
- How availability is measured and over what period.
- Excluded causes, maintenance rules and customer responsibilities.
- Severity definitions, response or restoration targets and escalation contacts.
- Claim evidence, deadline and the form of remedy.
- Whether credits are the only remedy and whether they cover consequential business losses.
Store a copy of the terms that applied when you purchased the service. Providers can change online documents, while your contract may incorporate a particular version.
Rank #3
- Capacity Display Variance: 500GB external ssd often appears as around 465GB on Windows. MacOS can show full 500 GB capacity. This is binary calculation difference and doesn’t affect SSD hard drive actual physical storage
- 1050 MB/s Speed: Instantly access to your files with blazing-fast 10Gbps external SSD read up to 1050MB/s and write up to 1000MB/s. LED Light indicates USB SSD instant activity
- Data Security: Solid state drives S.M.A.R.T. health diagnostics and adaptive TRIM optimizing data block management ensures consistent write speeds and extends the longevity of the portable SSD
- USB-C & USB-A Cable: Both cables featuring rapid USB 3.2 Gen2, this USB SSD effortlessly bridges devices, enabling seamless cross-platform file transfers and backup between computers, smartphones, tablets and iPhone
- Always Fast: No slowdowns for large file transfers. With SLC caching (25% of current available capacity allocated as high-speed cache), this external SSD delivers steady 10Gbps for transfers within the cache capacity
How do you diagnose and recover a GPU hardware fault?
Start by preserving evidence, then follow the platform’s documented recovery path. A reset or reboot may restore a GPU, but it cannot recreate work that was never checkpointed.
- Capture the incident. Record the instance identifier, region or zone, job ID, time of failure and recent changes before deleting or recreating anything.
- Inspect kernel logs. Google Cloud’s GPU VM troubleshooting guidance directs operators to check
dmesgor/var/log/kern.logfor numeric NVIDIA Xid errors. - Collect workload diagnostics. Preserve complete NCCL logs when distributed training is involved, health-check output, failing commands, diagnostic logs and the latest known-good configuration.
- Apply the documented recovery. Depending on the GPU instance family and fault, Google’s guidance may call for resetting the GPU or rebooting the VM. Follow the instructions for that instance type rather than assuming every GPU supports the same reset operation.
- Validate before restarting the full job. Confirm driver health, GPU visibility, interconnects, storage mounts and access to the latest checkpoint. Run a small test before committing to a long training run.
- Escalate with evidence. Give support the instance identifiers, suspected nodes, logs, health checks, failing commands and recent changes so the provider can distinguish a software fault from a host or hardware problem.
If the host cannot be recovered, launch a replacement instance only after confirming that the environment and checkpoint are available outside the failed machine.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How should you set recovery time and data-loss targets?
Define two targets for each workload. The recovery time objective (RTO) is how long the service can remain unavailable. The recovery point objective (RPO) is how much recent work or data you can afford to lose.
Match targets to the workload
- Interactive inference: A long RTO can become a customer-facing outage; plan a warm or quickly launchable alternative if requests are time-sensitive.
- Long training runs: RPO determines checkpoint frequency. Losing the last several hours of computation may be acceptable for one experiment and unacceptable for a deadline-driven model.
- Batch research: A longer RTO may reduce cost, but only if experiment metadata and intermediate results are durable.
- Regulated or contractual workloads: Evidence of backups, restore tests and assigned responsibilities may matter as much as the technical design.
NVIDIA’s Requirements for AI Clouds, version 2.4, calls for backups with restore-integrity verification, defined and auditable RTO and RPO, disaster-recovery and failover tests on a defined cadence with recorded results, and defined recovery roles. Those are requirements in NVIDIA’s named provider context, not an automatic promise to every cloud customer, but they are a useful baseline for evaluating a provider.
Rank #4
- NEARLY 2X FASTER THAN OUR PREVIOUS GENERATION(8) – move 1,000 high-res photos in under 60 seconds(6) with up to 2000MB/s transfer speeds(2).
- IP65 RATING AND UP TO 3M DROP PROTECTION(3) – protects against spills and drops.
- POCKET-SIZED – fits easily in pockets and small bags.
- SPACE TO OWN YOUR AI CONTENT – speed and capacity to download your high-res clips and photo edits.
- 256-BIT AES ENCRYPTION(4) – helps keep private files secure with password protection.
Which recovery strategy fits your risk and budget?
No strategy guarantees substitute capacity. Compare alternatives using RTO, RPO, GPU capacity, portability, data movement, operating complexity and contract coverage.
| Approach | What it provides | Main constraints |
|---|---|---|
| Same provider, different region | Potentially familiar APIs, images and billing with less environment change. | The incident may affect multiple regions; quotas, GPU type and network paths must be verified. |
| Second GPU provider | Reduces dependence on one provider and can supply an independent control plane. | Drivers, images, orchestration, identity, networking and GPU availability may differ; data transfer can be slow or costly. |
| Local or hybrid fallback | Provides an independently controlled place to run urgent jobs or hold copies. | Local hardware may lack equivalent GPUs, scale, cooling, staffing or current dependencies. |
| Cold rebuild from backups | Lowest ongoing infrastructure commitment when outages are rare. | Recovery is slow unless images, quotas, credentials and procedures are maintained and tested. |
Portability is more than downloading a model. You must be able to recreate the software environment, infrastructure configuration, data access, secrets, orchestration and required GPU capacity. Test a small workload on the proposed fallback; a plan that exists only in documentation has an unknown RTO.
What should you do during an active provider outage?
Use a written runbook so the incident does not become an improvised data-loss event.
- Confirm scope and status. Check the provider’s incident channel and your own monitoring, then determine whether the problem is compute, storage, networking, account access or a wider region.
- Freeze destructive actions. Do not terminate instances, overwrite checkpoints or rotate away credentials until you have captured evidence and confirmed which copies are safe.
- Identify the last durable state. Locate the newest verified checkpoint and its matching code, configuration and dataset version.
- Open a support case at the correct severity. Include timestamps, resource identifiers, logs, affected regions, business impact and the exact recovery you need.
- Activate the fallback. Use a pre-tested region, provider or local system only after checking GPU type, quota, networking, storage mounts and identity access.
- Validate the resumed workload. Compare loss, performance and output against the last known-good run before scaling up.
- Preserve the post-incident record. Keep logs, support responses, outage times and costs for any SLA claim and for improving the runbook.
Can NVIDIA Recovery fix a failed production workload?
Not as a production guarantee. NVIDIA’s NVIDIA Recovery documentation labels the project experimental and not ready for production use. It manages workloads rather than repairing hardware or storing checkpoints, and Kubernetes-native workload recovery is not implemented. Treat it as a project to evaluate against its current documentation, not as a substitute for durable storage and a tested failover design.
Best Value
- Capacity Display Variance: 1TB external ssd often appears as around 931GB on Windows. MacOS can show full 1 TB capacity. This is binary calculation difference and doesn’t affect SSD hard drive actual physical storage
- 1050 MB/s Speed: Instantly access to your files with blazing-fast 10Gbps external SSD read up to 1050MB/s and write up to 1000MB/s. LED Light indicates USB SSD instant activity
- Data Security: Solid state drives S.M.A.R.T. health diagnostics and adaptive TRIM optimizing data block management ensures consistent write speeds and extends the longevity of the portable SSD
- USB-C & USB-A Cable: Both cables featuring rapid USB 3.2 Gen2, this USB SSD effortlessly bridges devices, enabling seamless cross-platform file transfers and backup between computers, smartphones, tablets and iPhone
- Always Fast: No slowdowns for large file transfers. With SLC caching (25% of current available capacity allocated as high-speed cache), this external SSD delivers steady 10Gbps for transfers within the cache capacity
What mistakes make a cloud failure worse?
- Keeping the only checkpoint or dataset copy on an instance-local disk.
- Assuming an SLA percentage applies to GPU servers when it covers only storage or networking.
- Confusing a support response target with a guaranteed restoration time.
- Building a second-provider plan without testing drivers, containers, permissions, quotas and network access.
- Backing up files but never performing a complete restore.
- Ignoring data-transfer time, egress charges, access controls and encryption when moving to a fallback.
- Failing to assign who can declare an incident, approve failover and communicate with customers.
- Deleting failed instances before collecting logs and identifiers needed for diagnosis or an SLA claim.
How can you make a practical GPU-cloud continuity checklist?
Review this checklist quarterly and after any major change to your training stack:
- Map every dataset, checkpoint, artifact, secret, log and configuration to its storage location and retention policy.
- Mark which copies are outside the compute instance, outside the failure zone and, for critical workloads, outside the provider.
- Set an RTO and RPO per workload and document who approved them.
- Verify backup integrity and complete a restore, not merely a file listing.
- Keep reproducible images, dependency locks, infrastructure definitions and launch commands.
- Confirm substitute GPU models, quotas, networking and storage access in the fallback location.
- Test a restart and a cross-region or cross-provider recovery on a defined cadence; record the elapsed time and failures.
- Keep support contacts, severity rules, SLA claim deadlines and incident evidence requirements accessible when the control plane is impaired.
- Store an offline copy of irreplaceable artifacts if appropriate. An external SSD can be an additional backup copy, but it does not provide automatic replication, alternate compute or protection from a site-wide event when kept at the same location.
Frequently Asked Questions
How often should an AI team test its GPU-cloud failover plan?
Use a defined cadence that matches the workload’s RTO and RPO, and record the result each time. A test should include restoring data, recreating the environment, obtaining GPU capacity and running a representative job—not just checking that backups exist.
What evidence should I save for a GPU-cloud SLA claim?
Keep incident start and end times, affected resource identifiers, provider case numbers, status messages, logs and a description of the business impact. Also retain the SLA version and submit the claim by its stated deadline; NVIDIA’s general terms, for example, specify two months from incident start.
Is a second cloud account enough to protect against provider failure?
Not by itself. The second account must have independent access, quotas, stored artifacts, tested network paths and a documented way to recreate the workload; otherwise it may fail at the same time as the primary account’s control plane.
How do I estimate the cost of losing a training run?
Combine the compute time that must be repeated with staff time, delayed delivery, data-transfer or egress charges, and any contractual or customer impact. Use the result to decide whether more frequent checkpoints or an always-ready fallback is economically justified.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute
