Filtered Vector Search Benchmark: p99, Recall & TCO in 2026

Filtered vector search—not unfiltered QPS—is the production benchmark that reveals whether a retrieval system is ready for real users. At 1% or 0.1% tenant selectivity, recall can collapse, p99 latency can spike, and the architecture that looked cheapest on a benchmark sheet can become the most expensive to operate.

Key takeaways

• Filtered recall is the hard constraint. A system reporting 97% unfiltered recall may behave very differently when only 1% of the corpus is eligible.
• Memory dominates at scale. One hundred million 768-dimensional FP32 vectors require about 307 GB of raw storage before graph links, metadata, replicas, and headroom.
• Compare complete workloads. Cost per successful answer, p99 latency under saturation, freshness, shard fan-out, and recovery time matter more than headline QPS.

Why unfiltered benchmarks mislead

The clean ANN benchmark is easy to explain:

SELECT id, content
FROM documents
ORDER BY embedding <=> :query_embedding
LIMIT 20;

Every vector is eligible, so the index can follow its preferred search path. Production queries usually add constraints:

SELECT id, content
FROM documents
WHERE tenant_id = :tenant
  AND region = :region
  AND updated_at > :cutoff
ORDER BY embedding <=> :query_embedding
LIMIT 20;

Now the vector index and the predicate have to cooperate. If a tenant owns 1% of the corpus, globally nearest neighbors may be unusable. At 0.1% selectivity, the engine may need to inspect far more candidates, over-fetch aggressively, or route the query to a tenant-specific partition.

A filtered benchmark should vary selectivity at 100%, 10%, 1%, and 0.1%; test equality, range, array, and compound predicates; and measure top-k values of at least 10 and 100. It should also include concurrent writes, deletes, compaction, and exact-ground-truth recall, with p50, p95, and p99 latency measured at a defined saturation point.

The search strategy matters just as much. Pre-filtering, predicate-aware traversal, and post-filtering have different failure modes. A post-filtered search may find excellent global neighbors and then discard most of them. A pre-filtered search may preserve recall but lose the graph’s efficient navigation path.

Consider a request for 10 results against a predicate with 1% selectivity. A naive system may need to examine roughly 1,000 candidates to find 10 eligible documents, and often more because similarity and predicate membership are not independent. Predicate-aware indexing can reduce that work; correlated data can either help or hurt. The only reliable answer comes from measuring candidate counts and filtered recall on the target workload.

Published results need context

The following results are useful signals, not a single leaderboard. Hardware, concurrency, filters, and recall procedures differ—and several vendors did not publish all of those details.

The useful vendor request is specific: show p99 with your filters, top-k, write rate, and concurrency, then provide candidate counts and recall against exact search.

Memory, indexes, and the cost of scale

Raw vector storage is the first calculation, and it is commonly underestimated.

Memory Formula: Raw vector memory = number of vectors × dimensions × bytes per element

For FP32 embeddings:

These are decimal gigabytes: 1 GB equals 1 billion bytes. They include vectors only. HNSW links, identifiers, metadata, deleted-document tombstones, allocator fragmentation, caches, operating-system memory, query buffers, and replicas all sit on top.

For a 100-million-vector, 768-dimensional collection, two replicas require about 614 GB of raw vector memory. A practical HNSW deployment commonly needs roughly 1.3× to 2× raw vector memory per copy once graph storage, metadata, allocator overhead, and working space are included. The exact figure depends on implementation, M, vector precision, and whether vectors or graph pages live off-heap.

A rough sizing example makes the risk clear:

• Raw vectors: 307 GB
• HNSW and metadata overhead at 50%: about 154 GB
• Two copies: about 922 GB
• Headroom for caches, merges, failures, and growth at 25%: about 1.15 TB

That is a planning estimate, not a substitute for measuring the selected engine. It does show why a 512-GB machine cannot safely host “307 GB of vectors.”

INT8 reduces the vector portion to about one-quarter of FP32. Product quantization and binary formats can reduce it further, though many systems rerank candidates with higher-precision vectors to recover recall. Compare memory savings against filtered recall at the required top-k rather than relying on unfiltered accuracy.

GPUs, shards, and freshness

GPU acceleration can transform index construction. OpenSearch, for example, reported building a billion-vector, 1,024-dimensional HNSW index in 274 minutes using 44 GPU acceleration units. That does not make GPUs the cheapest always-on serving option. A frequently updated tenant corpus may not use enough GPU capacity to justify the cost and operational complexity.

Separate the workload into three measurements: initial index construction, peak query serving, and incremental updates or recovery. A design that wins the first test may lose the other two.

Sharding brings a similar trade-off. A distributed query usually routes to one or more shards, searches local indexes, sends candidates across the network, merges the global top-k, and may apply final filtering or reranking. With 20 shards per request, network and coordination can cost more than distance calculations. One slow shard also determines the p99.

Tenant-aware routing can avoid that fan-out when tenant sizes are manageable. Hard partitioning creates new problems, though: large tenants become hot spots, migrations become painful, and uneven traffic produces noisy neighbors. Test a single tenant on one shard, a tenant spread across many shards, mixed tenant sizes, and writes during search. Include node failure and measure recall while the system is degraded.

Freshness deserves its own metric. A successful database commit does not guarantee that the document is searchable immediately. Measure write-to-search visibility, especially when users expect newly created content to appear in retrieval results. Storage/compute-separated systems can lower idle cost, but cache misses, object-storage reads, autoscaling, and compaction may produce tail spikes.

TCO: cost per query is only the start

The useful economic metric is the cost of delivering a relevant answer, not the hourly price of an instance.

Cost Formula: Cost per million queries = monthly infrastructure cost ÷ (successful monthly queries ÷ 1,000,000)

Include compute, memory, replicas, storage, rebuilds, writes, deletes, network traffic, backups, disaster recovery, embedding generation, reranking, cache reads, and operations. A low-cost ANN query with weak filtered recall may require larger candidate sets, retries, or extra LLM context. The saving can disappear when measured per successful answer.

Databricks has reported 97% recall and 71 ms p99 for Lakebase Search on 100 million vectors, along with a claimed fourfold cost advantage over cloud PostgreSQL with pgvector in its comparison. AWS has reported query-charge reductions of up to 80% for S3 Vectors indexes above 10 million vectors. Those figures are useful starting points, not price guarantees; region, replication, traffic shape, storage tier, and freshness requirements change the result.

For smaller collections already living in PostgreSQL, pgvector or a managed PostgreSQL-compatible database may avoid synchronization pipelines. At larger scale, especially above 100 million vectors or at high concurrency, compare specialized engines such as OpenSearch, Milvus, Qdrant, or Pinecone with converged systems such as AlloyDB or Lakebase. Strict tenant isolation and 0.1% predicates usually call for tenant-aware partitions or predicate-aware indexing, not a generic ANN configuration.

The system to choose is the one that sustains filtered recall and p99 latency while tenants write data, shards recover, and traffic arrives unevenly. Unfiltered QPS is only the opening measurement.

Frequently Asked Questions

What is p99 latency for filtered vector search at 1% selectivity?

There is no universal p99. It depends on the index, predicate correlation, routing, concurrency, and whether the system pre-filters or post-filters. Measure 1% and 0.1% selectivity under saturated, production-like load rather than using unfiltered p99 as a proxy.

How much RAM does HNSW require for 100 million 768-dimensional vectors?

The raw FP32 vectors require about 307 GB. Graph links, metadata, replicas, tombstones, caches, and headroom can push a practical deployment toward 1 TB or more, depending on the implementation and replication plan.

Is filtered HNSW better than IVF or ScaNN?

Not universally. HNSW often suits mutable, memory-resident workloads, while IVF and ScaNN can reduce memory or improve large-scale throughput. Compare them at the same filtered-recall target, top-k, update rate, and p99 saturation point.

What is the best tenant-filtered vector-search architecture?

Use tenant-aware routing when tenant sizes are manageable and locality is strong. For uneven tenants or complex predicates, combine predicate-aware search with careful fan-out testing; partitioning alone can turn the largest tenants into hot spots.