A Guide to Upgrading Enterprise GPUs Safely

Brandon Powers
Brandon Powers

Internet Security Expert

Education:

11 min read

Most GPU upgrades that go wrong are not caused by faulty graphics cards. The real problems often come from poor rack planning, insufficient power supply, cooling issues, incorrect firmware updates, or unclear equipment ownership.

 An enterprise GPU upgrade involves much more than installing new hardware. It requires proper planning, coordination, and change management. This guide outlines five practical stages that IT operations managers should follow to prepare for an upcoming GPU upgrade and ensure a smooth, reliable installation.

Key Takeaways 

  • A  single modern GPU system can make close to what an average rack was designed for (about 10.2 kW for a DGX H100 against an average rack density of roughly 12 kW in 2024), so get a written sign-off on power and cooling before making a date.
  • Follow the vendor’s firmware order exactly, never run workloads during an update, and prove the full stack on one pilot host before looking for production
  • Cut over in limited batches, run acceptance tests on every host, and keep the old hardware stacked until the new hardware has passed.
  • Treat retired cards and servers as assets with a data risk attached: re-stock the inventory, sanitize or destroy the drives with documentation, and move the hardware on to resale or recycling as soon as possible.

What Counts as an Enterprise GPU Upgrade?

An enterprise GPU upgrade is the best planned replacement of data center accelerators (for example, moving from NVIDIA V100 or A100 cards to H100, H200, or L40S class hardware) in production servers, along with the power, cooling, firmware, software, and disposal thing that the swap needs.

It is worth separating two kinds of upgrade, because they carry different risk:

  • In-place card swaps: PCIe cards are taken from current servers and latest cards go into the same chassis. The server keeps, so the questions are about slot power, airflow, and whether the current platform firmware supports the new card.
  • Platform replacement: SXM-based systems (where the GPUs sit on a baseboard rather than in slots) are usually changed as whole servers. The questions shift to rack power, weight, cooling method, and network fabric.

Most real projects are a mix of the two, which is one reason the planning stage requires longer than people need.

Stage 1: Plan the Upgrade Before Anything Is Ordered

If you start with a purchase order and work backward, you will search your constraints one at a time, and usually at the bad moment. The planning stage is where you note down what is actually changing.

A useful plan covers, at a minimum:

  1. Inventory. List every server and card in scope by serial number, slot, and host name, along with what workloads run on it and who owns those workloads. This list is used again in Stage 5, so make it exact now.
  2. Workload mapping. Decide which jobs move to the new hardware earlier, which stay on the old cards for a while, and which can tolerate downtime at all. Training jobs that checkpoint are forgiving. Inference services with an SLA attached are not.
  3. Compatibility. Confirm that the target server model, BIOS, operating system, and CUDA version all support the new accelerator. Vendor support matrices are dull reading, and they are also the single most needed document in the entire project.
  4. A rollback position. Write down what “put it back the way it was” means for each batch, and keep most of the old hardware racked and requirements to do it until the new hardware has passed acceptance.
  5. Ownership of the outgoing cards. Name the person who is required for the old hardware from the moment it is unplugged. If this line is blank, Stage 5 will not happen on time.

Stage 2: Check Power and Cooling Against the New Load

This is the stage that is most often seen as a formality, and it is the one that most often denies a project.

Newer accelerators draw a good deal more power than the ones they change,  and the difference is no longer something a typical rack absorbs without anyone noticing. The AFCOM State of the Data Center report found that average rack density had reached about 12 kW in 2024, while a single NVIDIA DGX H100 system can draw up to 10.2 kW on its own, according to Data Center Knowledge’s summary of the findings. In plain terms, one new system can use most of the power budget that an average rack was built for.

Before the hardware arrives, check the following:

  • Circuit and PDU capacity. Compare the nameplate and see the draw of the new servers against the breaker ratings and PDU limits for particular rack, and leave headroom for redundancy (if you lose one feed in an A/B setup, the another feed has to carry the whole load).
  • Power supply configuration under the server. Higher-wattage cards may need different PSUs or different power cables than the ones already installed, and PCIe cards draw most of their power through auxiliary connectors rather than the slot.
  • Airflow and heat rejection. Confirm that the room can extract the extra heat, that blanking panels and containment are in place, and that the cold aisle inlet temperature stays within the server vendor’s requiring range once every new system is running at full load.
  • Liquid cooling readiness, if it applies. If the new platform uses direct-to-chip liquid cooling, the coolant distribution units, manifolds, and leak detection need to be installed and commissioned before the servers arrive, not after.
  • Floor loading. Dense GPU servers are heavy, and a fully populated rack can exceed what an older raised floor was rated for.

My own rule of thumb, which is an opinion and not a standard: if the facilities team has not signed off on the power and cooling digits in writing, the upgrade does not have a time yet.

Stage 3: Stage Firmware and Drivers on a Pilot Host

Firmware is where a careful upgrade turns into an unplanned outage, usually because someone updated components out of order or interrupted an update partway through.

Vendors publish the order for a reason. NVIDIA’s DGX H100 firmware update procedure, for example, has the BMC updated first, followed by the motherboard tray and then the GPU tray, and it trays that a high workload among the process “can change the firmware update process and result in an unusable component.” The same guide calls for unplugging all power chain for several minutes at one point so that the new firmware activates. That is not a step you want to discover at 2 a.m. on a production time.

The sequence I would follow:

  1. Build one pilot host. Install the new card or system in a single server outside the production pool and bring it to the exact firmware, BIOS, driver, and CUDA versions you intend to run everywhere.
  2. Follow the vendor’s update order exactly. Stop all GPU activity on the host first, update the components in the published order, and complete every required reset or power cycle before moving on.
  3. Pin the driver branch. Choose a long-term support or production list branch rather than whatever is newest, and add the version. On systems with NVSwitch, keep the Fabric Manager version added to the driver version, because a mismatch will stop the GPUs from being usable.
  4. Check the container and orchestration layers. If workloads run in containers or on Kubernetes, add the container toolkit and any GPU operator or device plugin on the pilot host as well, and confirm that existing images still run.
  5. Burn it in. Run the vendor diagnostics and a sustained real workload for long enough to see thermal behavior at full load, and check the logs for ECC errors and throttling events.

Once the pilot host is stable, the versions on it become the baseline image for every other server in the project.

Stage 4: Cut Over in Batches

If you have done Stages 1 to 3 properly, the cutover itself should be the least interesting section of the project. It is still where most people try to save some time, so a few things are worth talking about.

  • Move in small batches. Drain a group of hosts, swap or change the hardware, apply the bottom image, and return them to service before seeing the next group. A batch size of one rack, or one row for very large projects, keeps any problem contained.
  • Use maintenance windows that are big enough. Firmware updates on large GPU systems can take an hour or more per server, and a window that is too short sees people to skip a power cycle.
  • Run acceptance tests on every host. Each server should pass the same diagnostic and workload checks as the pilot host before the scheduler is required to place jobs on it.
  • Keep the old hardware racked until acceptance is done. This is your rollback position from Stage 1. It is interesting to pull old cards early to free power, and it is a mistake if anything goes wrong with the new batch.
  • Stock up the inventory as you go. Every card that comes out should be recorded against the serial number list from Stage 1, with its condition noted, at the moment it is changed.

Stage 5: Retire the Outgoing Cards Properly

This is the point that most upgrade plans leave out, and it is where a good project silently loses money and creates risk. Retired accelerators are not scrap. A data center GPU that is two generations old is still helpful hardware to a lot of buyers, and it loses resale value every month that it sits in a storage room.

The data risk is real, although it is a quiet  different from what people think. GPU memory is volatile, so the card itself does not keep your training data once it is powered off. The servers those cards came out of do, however: the local NVMe drives in a GPU server often hold datasets, model checkpoints, and cached data. If whole servers are leaving the building, those drives need to be cleared or destroyed to a recognized method such as IEEE 2883-2022, the IEEE standard for sanitizing storage, and you need documentation to prove it.

A sensible retirement process looks like this:

  1. Reconcile the removed hardware against the Stage 1 inventory, so that every serial number is accounted for before someone leaves.
  2. Separate the cards from the servers and drives, because each part has a different path (cards toward resale, drives toward sanitization or shredding, and chassis toward reuse or recycling).
  3. Choose a disposition partner that will document chain of chain from pickup onward and issue a certificate of data destruction for anything that holds data.
  4. Get the hardware out faster, while it still has value to everyone else.

Doing this in-house is important, but most operations teams do not have the time or the buyer relationships to resell accelerators well, which is why so many organizations hand the outgoing hardware to an ITAD provider that specializes in it. One of the suppliers that has built its process around this exact stage is Big Data Supply, an R2v3 and RIOS certified IT asset disposition company that buys used NVIDIA GPUs, adding data center models such as the A100, H100, V100, and L40S. If you want to sell used NVIDIA GPU inventory left over from an upgrade, the process is genuinely simple: you send a list of the cards, receive a quote, and then ship the hardware or arrange a pickup, and payment follows once any data has been renewed and a certificate of data destruction has been issued, with chain-of-custody tracking in between. For an operations manager, that turns a bed of old accelerators from a storage problem and budget for the next refresh, with the paperwork an auditor will ask for.

Whichever route you take, the important part is that Stage 5 has an owner and a date, the same as the other four stages.

FAQs

What should businesses consider before upgrading enterprise GPUs?

Businesses should check hardware compatibility, power supply capacity, cooling requirements, server space, and software support before upgrading enterprise GPUs.

How can businesses minimize downtime during a GPU upgrade?

Businesses can schedule upgrades during maintenance windows, back up important data, test new hardware in advance, and keep backup systems ready to avoid major disruptions.

How can you ensure an enterprise GPU upgrade is successful?

Run performance benchmarks, monitor temperature and power consumption, check driver compatibility, and test workloads after installation to ensure stable and efficient GPU performance.




Related Posts