H200 vs H100: Inference Cost & Performance Benchmark 2026

A 70B model may fit on two H200 GPUs once model weights and KV-cache headroom are included. The same workload can require four H100s—not because the model weights changed, but because 80 GB of memory per GPU leaves too little room for long contexts and concurrent requests.

That changes the economics. The H200 costs more per GPU-hour, but it may need half as many GPUs.

The right comparison isn’t the advertised hourly rate. It’s the cost of producing useful tokens at the latency and concurrency your application actually requires.

Cost formula: Cost per useful token = Total serving cost ÷ Successfully generated tokens

H200 vs. H100 pricing

The following figures use the supplied 2026 reference rates. AWS figures are based on an AWS Capacity Block reference configuration in the US market, priced per GPU equivalent; the exact AWS region, block term, host configuration, and availability should be confirmed before committing capacity. CoreWeave and Lambda figures are separate B200 reference rates, not apples-to-apples alternatives to the AWS entries.

*Monthly figures assume 730 hours of continuous GPU allocation. They exclude CPUs, host memory, storage, networking, orchestration, support, taxes, data transfer, and idle capacity.

The AWS H200 rate is about 45% higher than the H100 rate. If both configurations use the same number of GPUs, H200 must produce roughly 1.45 times as many useful tokens to break even.

That isn’t the only way to win. Memory capacity can reduce the GPU count:

• Four H100s: 4 × $4.72 = $18.88 per hour
• Two H200s: 2 × $6.865 = $13.73 per hour

The two-H200 configuration is about 27% cheaper on GPU capacity alone. It also reduces the number of GPU-to-GPU communication paths and may simplify deployment.

The saving isn’t guaranteed. Host memory, CPU allocation, interconnect topology, networking, and provider packaging can change the result. A two-GPU H200 instance with an unusually large host configuration may cost more than a tightly packaged four-GPU H100 instance. Compare complete instances and monthly commitments, not just GPU line items.

Why H200 memory matters

Inference memory goes well beyond model weights. The KV cache grows with context length and active sequences, so memory pressure increases with both prompt size and concurrency.

A 70B model that technically fits on H100 may still perform poorly if only a small amount of memory remains for the KV cache. Operators then reduce batch size, use CPU or NVMe offload, increase tensor parallelism, or accept out-of-memory failures during traffic spikes.

H200’s 141 GB of HBM is useful for:

• 32K- or 64K-token contexts
• Larger continuous batches
• 70B- to 120B-class models
• Retrieval-augmented prompts with large documents
• Multi-turn agent sessions
• Serving more than one model on a host

Its 4.8 TB/s memory bandwidth also helps decode-heavy workloads, but it doesn’t guarantee a 43% performance improvement over H100. Kernel efficiency, quantization, attention implementation, interconnects, and the request mix all matter.

A basic vLLM deployment might look like this:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching

The useful comparison runs the same model, quantization, context limit, prefix-caching policy, and request trace on both GPU types. Record:

• Input and output token counts
• Time to first token
• Inter-token latency
• Tokens per second
• p95 and p99 request latency
• GPU memory and utilization
• Failed, cancelled, or timed-out requests
• Total infrastructure cost per successful request

Peak decode speed is not enough. If a faster configuration misses the p99 target and causes retries or extra replicas, its theoretical throughput won’t lower the bill.

Where H100 still wins

H100 remains the sensible choice when the model fits comfortably within 80 GB and the workload has predictable context lengths. It also benefits from broad CUDA support, mature deployment tooling, and a large installed base.

The premium for H200 becomes easier to justify when H100 requires:

• CPU or NVMe offload
• Four-way tensor parallelism instead of two-way
• Aggressive quantization to preserve KV-cache space
• Very small batches
• Separate replicas for traffic spikes
• Frequent out-of-memory protection or restarts

B200 belongs in the same analysis, but its higher memory capacity and bandwidth don’t make it automatically cheaper. Compare B200 and H200 with the same model, precision, context length, concurrency, and service-level objective. Provider-specific host packaging and network fabrics can matter as much as the GPU specification.

The database can dominate request cost

The GPU is only one part of an inference request. A typical application may retrieve conversation state, permissions, documents, feature data, and tool results before generation begins.

For vector storage, a 1,536-dimensional FP32 embedding consumes 6,144 bytes. Ten million vectors require about 61 GB of raw vector data; 100 million require roughly 614 GB. IDs, metadata, replicas, WAL, caches, and an approximate-nearest-neighbor index add substantially more.

HNSW overhead varies by implementation and settings, but planning for roughly 1.3× to 2× the raw vector payload is safer than sizing from the embedding data alone. A 64 GB PostgreSQL node can run out of memory while building an index even when the raw vectors appear to fit.

pgvector keeps retrieval close to authorization and transactional data, which often simplifies correctness. Its trade-offs appear in index memory, vacuum behavior, write amplification, and query plans.

ClickHouse is attractive for large analytical workloads and batch ingestion, but distributed retrieval can introduce fan-out and tail latency. Cold parts, compaction, poor selectivity, and concurrent scans can turn an apparently fast query into a slow one.

For inference economics, the key question is whether database latency forces the GPU to wait. A 100-millisecond retrieval step can erase the benefit of a modest improvement in decode speed, especially for short responses. Measure the complete request path rather than optimizing the GPU in isolation.

Runtime choice: Rust or Go?

Rust can reduce memory use and tail latency in CPU-heavy gateways, parsers, serializers, and high-rate network services. Go is often the better operational choice for I/O-bound APIs because development is faster and the ecosystem is broad.

Start with a profile, not a language preference. If a Go service spends 80 ms waiting on a database and 5 ms in application code, rewriting it in Rust won’t materially improve user-facing latency. If it handles thousands of requests per second and spends most of its time allocating, parsing, or serializing, Rust may improve p99 behavior and reduce CPU cost.

Evaluate Rust when CPU utilization is high, the p99 target is below roughly 10–20 ms, and profiling identifies allocation or garbage-collection pauses in the critical path. Stay with Go when external services dominate latency or maintainability and delivery speed matter more than a small runtime improvement.

A practical purchase decision

Choose H100 when:

• The model fits with substantial memory headroom.
• Context lengths and concurrency are moderate.
• The provider offers materially better availability or committed pricing.
• You need the lowest raw GPU-hour cost.

Choose H200 when:

• KV-cache pressure limits H100 throughput.
• H100 requires offload or extra tensor-parallel shards.
• Long context and concurrent sessions are central to the workload.
• Two H200s can replace four H100s without missing latency targets.

Choose B200 when its throughput, memory bandwidth, or model-specific performance justifies the provider’s complete instance price.

For every option, run a controlled comparison using the same workload and report tokens per dollar, not just tokens per second. Include the real region, capacity-block or on-demand term, instance shape, host resources, and expected utilization in the cost model.

Frequently Asked Questions

Is H200 worth paying more than H100 for LLM inference?

It is when the extra 61 GB of HBM reduces tensor parallelism, CPU offload, or KV-cache pressure. If the model fits comfortably on H100 and meets its throughput target, H100 will usually have the lower raw capacity cost.

How do H200 and H100 compare on tokens per dollar?

Divide total serving cost by successfully generated tokens at the same context length, concurrency, precision, and latency target. At the reference AWS rates, H200 needs about 1.45 times the useful output of H100 when both configurations use the same number of GPUs.

How much does an H100 cost per month?

At $4.72 per GPU-hour and continuous 730-hour usage, the GPU capacity costs approximately $3,446 per month. Host resources, networking, storage, support, taxes, and idle time can raise the actual bill.

When should a team move from pgvector to another vector database?

Move when index memory, vacuum behavior, write amplification, analytical scans, or retrieval tail latency becomes a production bottleneck. Vector count alone isn’t decisive; a well-designed pgvector deployment can support millions of vectors when PostgreSQL remains the simplest system that meets the workload.