Enterprise vector database: when you need one and when you do not

Every enterprise AI project that touches internal documents reaches the same question: where do the embeddings go? The reflex is to add a dedicated vector database, and sometimes that is right. Often it is premature, and the real determinant of whether retrieval works is somewhere else entirely. This article covers what a vector database does, when a dedicated one is justified, and why most retrieval failures have nothing to do with the database.

What a vector database does

A vector database stores high-dimensional vectors (embeddings) and answers one core question fast: given a query vector, which stored vectors are nearest to it? Nearness in embedding space approximates semantic similarity, so "find the vectors closest to this question" becomes "find the passages most relevant to this question." That is the retrieval step behind semantic search, recommendation and retrieval-augmented generation (RAG).

Doing this exactly across millions of vectors is too slow, so vector databases use approximate nearest neighbor (ANN) search. The dominant index is HNSW (Hierarchical Navigable Small World), a graph structure that trades a small, tunable loss of recall for a large gain in speed. IVF (inverted file) indexes are the other common family. The practical point is that ANN is a speed-versus-recall tradeoff you configure, not a solved absolute.

A vector database also does what a plain index does not: it stores metadata alongside each vector and filters on it (by tenant, date, document type, access level) as part of the search. In an enterprise setting, that metadata filtering is often more important than raw search speed.

When you do not need a dedicated one

The most common mistake is reaching for a specialized vector database before the problem demands it. For a large share of projects, the pragmatic answer is pgvector: an extension that adds vector columns and ANN indexes to a PostgreSQL you already run.

pgvector handles millions of vectors comfortably, supports HNSW indexing, and keeps everything in one system, which means one backup strategy, one access model, one operational surface. The metadata you filter on usually lives in relational tables already, so a join replaces a cross-system lookup. For teams whose corpus is in the tens or low hundreds of thousands of documents, this is not a compromise, it is the correct architecture.

Starting here has a second advantage. It defers the decision until you have real usage data (query volume, latency requirements, corpus growth) to justify a dedicated engine, instead of guessing at scale you do not yet have.

When a dedicated engine earns its place

A dedicated vector database (Qdrant, Weaviate, Milvus, or managed services like Pinecone) becomes justified when specific pressures appear:

• Scale. Corpora in the tens of millions of vectors and up, where a general-purpose database's vector index stops being the right tool and a purpose-built one pays off.
• Throughput and latency. High query concurrency with tight latency budgets, where a specialized engine's indexing and caching matter.
• Filtered and hybrid search at scale. Combining vector similarity with structured metadata filters and keyword (BM25) ranking, efficiently, over large collections. Dedicated engines optimize exactly this.
• Horizontal scaling. Sharding and replication designed for vector workloads, rather than bolted onto a relational system.

The decision should follow evidence. Adopting a distributed vector database for a hundred thousand documents adds an operational system to run, secure and back up, in exchange for headroom you are not using. Adopting one for fifty million vectors under load is simply the right tool.

Sovereignty and where the vectors live

For European enterprises, the retrieval layer is also a data-residency question. Embeddings are derived from your source documents, and depending on the content they can be sensitive or personal data in their own right. Sending them to a managed vector service outside the EU, or generating embeddings through an external API, moves that data across a boundary that GDPR and internal policy may not permit.

This is why self-hostable engines (Qdrant, Weaviate, Milvus) and in-database options (pgvector) matter beyond cost. They keep both the embeddings and the retrieval inside a controlled, European environment, which is the same reasoning that drives private, self-hosted language models. The retrieval layer and the model layer share one requirement: the sensitive data stays where you control it.

Why most retrieval failures are not the database

When a RAG system returns wrong or irrelevant answers, the database is rarely the cause. The quality of retrieval is decided upstream, and a fast index over bad inputs just returns bad results faster.

• Chunking. How documents are split determines what can be retrieved. Chunks too large dilute relevance; too small lose context. This single choice affects answer quality more than the choice of database.
• Embedding model. The embedding model decides what "similar" means. A model poorly matched to your domain or language (a concern for German and multilingual corpora) produces a space where the wrong things are near each other.
• Metadata filtering. Without filtering by access level and tenant, retrieval can surface documents a user should not see, which is a security failure, not just a relevance one.
• Evaluation. Teams that ship retrieval without an evaluation set cannot tell whether a change helped or hurt. Retrieval quality has to be measured against known-good question-answer pairs, or tuning is guesswork.

The order of investment should follow this: get chunking, embeddings, filtering and evaluation right first, on the simplest storage that works, and choose the database based on the scale you actually reach.

Failure patterns

From the field, the patterns that make vector retrieval underperform:

1. Dedicated database first. A distributed engine adopted for a corpus pgvector would have handled, adding operations for unused scale.

2. No evaluation set. Retrieval shipped without a way to measure relevance, so every later change is a guess.

3. Chunking as an afterthought. Documents split naively, capping answer quality no matter how good the model or index.

4. Embeddings across a border. Sensitive content embedded through an external API or stored in a non-EU service, creating a residency problem.

5. No metadata filtering. Similarity search without access-level filters, surfacing documents to users who should not see them.

Talk through your retrieval architecture

DNA Solutions helps European enterprises build the retrieval layer behind semantic search and RAG: the right storage for the scale you actually have, self-hosted and EU-resident where sovereignty requires it, with chunking, embeddings, filtering and evaluation treated as the parts that decide quality. Whether you are starting with pgvector or scaling to a dedicated engine under load, we size it to the problem rather than the hype. Talk to us.