H100 vs H200: 2026 Real-World GPU Inference Cost Benchmark

A 70B model can fit on one H100—until production begins.

The quantized weights may occupy roughly 35–45 GB, leaving apparently comfortable headroom in an 80 GB H100. Add a 32K or 128K context window, KV cache, CUDA graphs, runtime buffers, and concurrent requests, and that headroom can disappear quickly. The deployment may need two H100s for tensor parallelism.

An H200 SXM GPU, with 141 GB of HBM3e, can often keep the same workload on one card. That changes the economics more than the small difference in hourly rental price.

The H100 is usually cheaper per GPU-hour. The H200 can be cheaper per completed request or delivered token when its extra memory avoids multi-GPU sharding, supports more concurrent sequences, or keeps long-context traffic inside the latency target. This is a cost model, not a claim of matched benchmark performance: actual results depend on the model, quantization, runtime, traffic pattern, and service-level objective.

Key takeaways

• Representative pricing is about $6.16 per H100-hour versus $6.31 per H200-hour.
• The H200’s 141 GB of HBM3e can matter more than its raw compute advantage, especially for 70B models and long contexts.
• Compare cost per completed request or blended token at a defined p99 latency—not simply dollars per GPU-hour or peak tokens per second.

Hourly price is only the starting point

A representative October 2026 pricing snapshot looks like this:

*730 hours per month, GPU charges only. Figures are rounded from the hourly rates.

At the example CoreWeave rates, the H200 is only about 2.4% more expensive than the H100. That makes the decision look easy until the deployment requirements are included: host CPU, system memory, storage, networking, NVLink topology, reservation terms, egress, and available capacity.

An eight-GPU H100 machine at $6.16 per GPU-hour costs about $35,977 per month before those extras. At $6.88 per hour, it costs approximately $40,178. A production budget may add another 15–35% for the rest of the platform.

The H200’s advantage is capacity, not a guaranteed twofold improvement in generation speed. An 80 GB H100 handles many 7B and 13B models comfortably. A 70B model in 4-bit precision may use 35–45 GB for weights alone. That estimate does not include:

• KV cache;
• activations;
• CUDA graphs and runtime buffers;
• batching overhead;
• long prompts;
• concurrent sequences.

A weights-only fit is not a production fit. If a 70B service needs two H100s to meet its context and concurrency requirements, two H100s cost $12.32 per hour at the $6.16 rate. One H200 costs $6.31 per hour. Even if the H200 delivers similar raw generation speed per GPU, the single-card deployment can have a much lower infrastructure bill and simpler serving topology.

The capacity figures above refer specifically to an H200 SXM configuration. Other H200 form factors or provider offerings may differ in memory, host design, interconnect, and pricing. Confirm the exact hardware rather than comparing product names alone.

A better measure: tokens and requests per dollar

For a continuously busy GPU producing output tokens, the basic calculation is:

Cost per million output tokens = GPU-hour price × 1,000,000 ÷ (output tokens per second × 3,600)

At $6.16 per H100-hour:

These figures assume stable generation and full utilization. Real services have bursts, uneven prompt lengths, model loading, deploys, retries, and periods with no queued work.

If the GPU generates 3,000 tokens per second while active but is useful only half the time, its effective rate is 1,500 tokens per second and the cost doubles. A more realistic utilization-adjusted measure is:

Effective output tokens per dollar = sustained tokens/sec × useful utilization ÷ GPU-hour price

For example, 3,000 tokens per second at 70% useful utilization on a $6.16 H100 produces an effective 2,100 tokens per second. That equals about 1.23 million output tokens per dollar.

The formula measures output tokens. Production economics often need a blended measure that includes input and output tokens, because long prompts can consume substantial prefill compute and KV-cache memory. For many applications, cost per completed request is even clearer:

Cost per completed request = total serving cost ÷ successful requests completed within the latency target

Track prompt tokens, generated tokens, request duration, and rejected or timed-out requests. A GPU that looks efficient on output tokens may be expensive if it produces long time-to-first-token values or misses the p99 target.

When the H200 premium pays off

The H200 is most attractive in three situations.

Long-context 70B serving

Each active sequence consumes KV-cache memory. A long-context assistant can exhaust an H100 even when the quantized weights fit comfortably. Once the cache fills, the scheduler may reduce concurrency, spill work across GPUs, or reject requests.

The H200’s larger memory pool gives the serving engine more room for concurrent sequences and longer prompts. That can improve both throughput and p99 latency, but only if the workload is memory-constrained. A compute-bound model with short prompts may see little benefit.

Avoiding tensor parallelism

Two GPUs do not automatically provide twice the useful performance. Tensor parallelism adds communication and synchronization, and its cost depends on the model, interconnect, framework, and batch shape.

At the representative CoreWeave rate:

The H200 premium over one H100 is about $109 per month in this example. The saving versus two H100s is roughly $4,388 per month before host-level differences.

Simpler operations

A single-GPU deployment is easier to schedule, autoscale, drain, observe, and recover. It also avoids some placement and networking constraints when capacity is tight. Those benefits rarely appear in a GPU-only spreadsheet, but they matter during rolling upgrades and incident recovery.

H100 remains the sensible choice when the model fits with comfortable memory headroom, traffic is high enough to batch efficiently, and the workload is primarily compute-bound. Paying for unused H200 memory does not improve economics.

What a useful comparison should measure

Do not present a price-sheet comparison as a performance benchmark. A matched test should hold the following constant:

• model and checkpoint;
• quantization format;
• inference engine and version;
• input and output token distributions;
• concurrency and arrival pattern;
• sampling settings;
• latency target;
• warm-up and test duration.

Measure time to first token (TTFT), inter-token latency, p50/p95/p99 request latency, sustained output tokens per second, completed requests, KV-cache occupancy, GPU memory, power, and model-load time.

A practical test might use 4,096 input tokens, 512 output tokens, concurrency levels of 32, 64, and 128, and a 15-minute steady-state run. Report both raw throughput and the amount of work completed inside the p99 service-level objective.

That distinction matters. Offline throughput can make a GPU look excellent while interactive users wait too long for the first token. Conversely, a lower-throughput card may win if its memory capacity prevents sharding and keeps tail latency stable.

Keep the RAG cost model in view

The GPU is only one part of an inference request. Retrieval, authorization, conversation-state reads, and document filtering can dominate latency before the model is saturated.

For a transactional RAG system with tens of millions of embeddings, PostgreSQL with pgvector is often attractive because vectors remain close to tenant filters, permissions, and application records. Memory planning is critical. Ten million 1,536-dimensional float32 vectors require:

10,000,000 × 1,536 × 4 bytes = 61.4 GB of raw vector data

That excludes HNSW indexes, table overhead, replicas, and working memory. If the index competes with the buffer pool, retrieval p99 can rise sharply.

As an initial planning range, a warm, filtered pgvector query may target 50–150 ms p99. Memory-constrained or highly concurrent workloads can move into the 150–500 ms range. These are planning targets, not universal guarantees; replay real tenant filters, top-k values, update rates, and concurrency before changing databases.

ClickHouse can make more sense when vector retrieval is combined with large event-history scans or append-heavy telemetry. It is not automatically a low-latency transactional vector store. If retrieval adds 300 ms to a request, saving a few cents per million generated tokens may not improve the product.

Frequently Asked Questions

Is the H200 cheaper than the H100 for 70B inference?

Not by hourly price. In the example above, one H200 costs $6.31 per hour versus $6.16 for an H100. The H200 can be cheaper overall when its 141 GB of memory avoids two-GPU sharding, supports more KV cache, or keeps long-context traffic within the p99 target.

How should I calculate inference cost?

Start with GPU-hour price divided by measured output tokens per second, then adjust for useful utilization. For production, also calculate cost per completed request or blended input-plus-output token, including host infrastructure and idle capacity.

Does more H200 memory always improve performance?

No. It helps when memory limits concurrency, context length, or model placement. If the workload is compute-bound and already fits comfortably on an H100, the extra capacity may not justify the premium. Benchmark both cards with the real model and traffic pattern.