The NVIDIA H100 is the Hopper-generation datacenter GPU most clouds still rent first. If you are about to pay for one, five numbers matter: memory capacity, memory bandwidth, dense FP8 and BF16 throughput, NVLink and power. Everything else is a footnote.
These are the figures I use. The same table is on the H100 page with the live listings. For the form factor differences, read H100 SXM vs PCIe vs NVL first.
What is the H100?
The H100 is NVIDIA’s Hopper GPU for AI training and inference, built on TSMC 4N with about 80 billion transistors. It introduced the Transformer Engine and FP8 to the mainstream, and it is the card most 70B-class work is sized around. The next steps up are H200, with more HBM3e, and Blackwell with B200.
H100 VRAM and memory bandwidth
- Capacity: 80 GB HBM3 on SXM and PCIe. NVL carries 94 GB per GPU.
- Bandwidth: about 3,350 GB/s on SXM, about 2,000 GB/s on PCIe, about 3,938 GB/s on NVL.
- Bus width: 5,120-bit on the SXM module.
- ECC: yes.
H100 VRAM is the number people search for, and 80 GB is the wall. Weights, optimizer states and KV cache all live there. A 70B model at 16-bit needs about 140 GB for weights alone, so it does not fit one H100. At 4-bit it needs roughly 35 to 40 GB for weights and fits with room for cache. Memory bandwidth decides how fast tokens come out when you decode, which is why H100 SXM feels so different from PCIe.
H100 TFLOPS: dense vs sparse
- FP8: about 1,979 TFLOPS dense (3,958 with sparsity).
- FP16 / BF16: about 989.5 TFLOPS dense (1,979 with sparsity).
- TF32: about 494.5 TFLOPS dense.
- FP32: 67 TFLOPS. FP64: 34 TFLOPS, 67 with Tensor Cores.
NVIDIA’s headline numbers are often with sparsity, and your trainer does not use sparse kernels unless you wrote them. Dense FP8 on SXM is about 1,979 TFLOPS, not 3,958. Mixing the two is how a datasheet turns into a bad capacity plan.
H100 NVLink, PCIe and power consumption
- Interconnect: NVLink at 900 GB/s on SXM; a 600 GB/s bridge on PCIe and NVL.
- Host interface: PCIe 5.0 x16 (128 GB/s).
- Board power: up to 700 W on SXM, around 350 W on PCIe, 350 to 400 W on NVL.
- Partitioning: MIG lets you split one card into up to seven instances.
H100 power consumption is a hosting question as much as a spec. A 700 W SXM module needs a real node and real cooling. A marketplace host that looks cheap is often a chassis that was never meant to hold that load. Read the TDP on the listing, not just the dollar.
What fits in one H100?
As a rule of thumb, one H100 runs models up to roughly 120B parameters at 4-bit quantization, or about 31B at 16-bit, assuming a 32K context. Longer context, bigger batches and fat optimizer states all take memory from that budget. If you can name the bytes that spill, H200 and its 141 GB is the upgrade. If you cannot, it is a more expensive H100.
H100 price and where to rent one
Specs do not tell you the hour. The H100 on-demand price per hour varies by form factor, provider type and region. See the H100 listings, the GPU pricing chart and how to rent an H100 on demand.
FAQ: H100 specs
How much VRAM does the H100 have?
80 GB of HBM3 on the SXM and PCIe cards. The NVL variant has 94 GB per GPU.
What is the H100 memory bandwidth?
About 3.35 TB/s on SXM, about 2 TB/s on PCIe and about 3.9 TB/s on NVL.
How many TFLOPS is an H100?
Dense FP8 is about 1,979 TFLOPS and dense BF16 about 989.5 TFLOPS on SXM. The larger numbers you see on slides are with sparsity.
How much power does an H100 use?
Up to 700 W on SXM. PCIe sits around 350 W.
H100 vs A100: is it worth it?
If you use FP8 and the Transformer Engine, yes. If the job is Ampere-shaped, A100 can still be the cheaper hour. See H100 vs A100.
Some outbound links in these articles may be affiliate links. Rankings on GPUBeacon stay independent.