Versioned Search

VectorLink is the name of the embedded semantic search engine that treats your data the same way TerminusDB does: as a versioned, branchable history of commits. Every commit can be indexed automatically and becomes an independent, reproducible snapshot. You search a specific commit and always get the same results, regardless of what was indexed afterwards. Branches share their parent's vectors instead of recomputing them. Every version with complete hybrid search.

TerminusDB standardized on a small 8-bit quantized multilingual model (vectorlink/multilingual-e5-base) which is the TerminusDB default model for data products. Standardization enables interoperability, at the cost of perfect embeddings, a hard tradeoff. It can of course be changed to any other model, but we instead recommend to use slot 2 of VectorLink for your custom model. This enables both low fidelity and high fidelity search in the same data product. Use what makes most sense for the use case, but find vector similarity across different data products. This means every chunk fits in exactly 768 bytes of vectors, making each vector reasonably small.

This website uses VectorLink to provide semantic search across all documentation pages and naturally uses the default model, with fully self-sovereign vector production on the same dynamically scaling TerminusDB infrastructure. Leveraging the historic search capabilities we have is of course a next step also for the site.

The engine — VectorLink 2.0 — is built in Rust based on portions of LanceDB. It is the successor to the earlier VectorLink indexer, with a maintained, columnar vector store that adds full-text search and hybrid retrieval out of the box. Embeddings come from a configurable provider, defaulting to a local CPU model so the whole stack runs offline.

Why versioned search matters

Traditional vector stores flatten your data into a single index. When a document changes, the old version is gone. You cannot ask "what did search return at this commit?" — the answer has been overwritten.

Versioned Search takes a different approach. Each commit is a tagged, immutable snapshot. Searching commit C1 always returns the same results, even after you index C2, C3, and beyond. This means:

• Reproducibility — audit what search returned at any point in history.
• Branch isolation — a feature branch can index experimental documents without polluting the main branch's search results.
• No recompute on unchanged data — indexing a new commit only embeds what changed. Documents that stayed the same are reused from the parent snapshot.
• Staleness without blocking — if you search a commit that hasn't been indexed yet, the engine serves the nearest indexed ancestor and tells you what it actually served via a response header. It never blocks, and it never silently serves a newer-than-requested snapshot.

How it fits with TerminusDB

TerminusDB owns the data, the schema, and the commit history. Versioned Search owns the embeddings and the vector store. The boundary is clean:

• TerminusDB renders documents to text (via automatic GraphQL + Handlebars templates), computes the diff between two commits, and pushes the delta to the search engine.
• The search engine embeds the text, stores vectors versioned per commit, and answers search queries.
• TerminusDB fronts search requests — it authorises the caller, then proxies to the engine. The engine itself is a trusted component behind a shared admin secret, not directly exposed to end users.

The engine can run in-process inside TerminusDB as a builtin plugin — set TERMINUSDB_INDEXER_BACKEND=vectorlink and no separate service is needed — or as a remote service reached over HTTP. Both modes share the same routes and indexing protocol; see Embedded VectorLink Engine for the trade-offs.

In production, TerminusDB drives indexing automatically. You can also drive the engine directly via API/curl for testing, debugging, or standalone use.

What you can do with it

• Semantic search — find documents by meaning, not just keywords. "Wise old Jedi master" finds Yoda even if those exact words never appear in his description.
• Full-text search — exact keyword matching over the rendered text. Useful for identifiers, rare tokens, and precise lookups.
• Hybrid search (default) — combines vector and full-text results using reciprocal-rank fusion. Best general-purpose relevance.
• Similar documents — given a document, find its nearest neighbours in the same snapshot.
• Duplicate detection — surface near-duplicate groups across a population, or across two record sets for entity resolution.
• Typeahead suggestions — fast full-text-only autocomplete for UI search boxes.
• Per-commit snapshots — search any commit in history and get reproducible results.
• Branch-aware indexing — branches share parent vectors; each branch sees only its own lineage.
• Automatic asymmetric model support — the default model (multilingual-e5-base) uses different text prefixes for indexing versus querying. The engine applies the correct prefix automatically based on the operation context, so you never have to manage passage: / query: prefixes by hand. Getting this wrong is the most common silent retrieval quality killer with asymmetric models, so the engine eliminates the footgun entirely.

Search modes at a glance

Results are ranked by distance in [0, 1]: 0 is identical, 0.5 is unrelated, 1 is opposite. Smaller is closer.

Where to go next

• Concepts — the vocabulary: domains, commits, branches, chunks, distance.
• Quickstart — bring up the stack, index a commit, run a search in five minutes.
• Embedded Engine — run VectorLink inside TerminusDB as a builtin plugin; configuration, storage, and choosing embedded vs remote.
• How automatic indexing works — the three things TerminusDB needs before it indexes by itself, what to expect from the first search, and a troubleshooting list.
• TerminusDB Push Indexing — how TerminusDB automatically indexes documents via store_indices, with a worked example across three commits.
• Searching — GET vs POST, modes, filters, pagination, and reading the response.
• Clustering Embeddings — a second embedding role for near-duplicate grouping and entity resolution.
• History & Branching — per-commit snapshots, branch-out with block reuse, and staleness handling.
• Entity Resolution — tuning duplicates and cross-set matching for record linkage.
• API Reference — the full HTTP endpoint reference.

Using search as memory for AI agents? See A Knowledge Graph for AI Agents for a branch-per-run workflow.