R7.05 · Data Modeling & Memory — module hub

Vectors & similarity search.

Store vectors and find similar items using Redis 8's native Vector Sets—an HNSW-based data structure supporting semantic search, RAG, recommendations, and classification with optional filtered queries.

A Vector Set is a Redis 8 data type where each element carries a vector instead of a score. VADD inserts; VSIM returns the nearest neighbours in O(log N) via an automatically-built HNSW graph. This is a standalone Redis 8 example (the BCS stack reaches the same algorithm via the valkey-search module — the engine note below). Select a use-case; the readout names the query it maps to.

Six use-cases for a vector set. Select one; the readout names the query pattern and the command that drives it.

readout · Select a use-case. A vector set answers “what is like this?” — a semantic neighbourhood, not an exact match. The six use-cases are all VSIM queries: semantic search, RAG retrieval, recommendations, classification, deduplication, and anomaly detection.

§1

When to use vector sets

vector-sets.md.txt · When to Use

A vector set answers the question “what is semantically like this?” — a question no sorted set, hash, or keyword index can answer. The inputs are numerical embeddings produced by a machine-learning model; similar items have vectors close together in the embedding space. Six concrete applications follow from that one query shape:

Semantic search — find documents or products by meaning, not keywords. A user query becomes an embedding; the nearest neighbours are the relevant results.
RAG (Retrieval-Augmented Generation) — ground an LLM response in your private data by retrieving the most relevant document chunks before generating.
Recommendations — “users who liked X also liked…”: VSIM key ELE product:123 returns the nearest items by embedding.
Classification — store labeled examples; assign a category to a new item by majority vote among its k nearest neighbours.
Deduplication — find near-duplicate content; items with cosine score near 1.0 are semantically near-identical.
Anomaly detection — items far from their nearest neighbours (low similarity scores) are anomalous.

A vector set answers a semantic neighbourhood query. Every use-case above is a variation on VSIM returning the nearest neighbours of a query vector or of an existing element.

§2

Core commands

vector-sets.md.txt · Core Commands

Adding elements and querying for neighbours are the two primary operations. All other commands are introspection and maintenance:

VADD key VALUES 3 0.1 0.5 0.9 my-element    # add an element with an inline vector
VADD key FP32 <binary-blob> my-element      # …or a binary blob (faster for clients)

VSIM key VALUES 3 0.1 0.5 0.9 COUNT 10 WITHSCORES   # 10 nearest by a query vector
VSIM key ELE existing-element COUNT 10 WITHSCORES   # …or nearest to an existing element

VCARD key            # count elements
VDIM  key            # get vector dimension
VEMB  key element    # retrieve an element's vector
VREM  key element    # true deletion, memory reclaimed

Options on VADD: Q8 (default — 8-bit quantization, 4× memory reduction, ~96% recall) · BIN (1-bit, 32× reduction, ~80% recall) · NOQUANT (fp32, best recall) · M num (HNSW connectivity, default 16) · SETATTR '{...}' (JSON metadata for filtered search). Options on VSIM: COUNT n, WITHSCORES, EF num (exploration factor), FILTER expr.

The HNSW graph is built automatically as elements are added. No FT.CREATE step, no separate index build phase, no schema declaration — VADD inserts directly into the live graph. This is the Redis 8 surface; the Valkey path is different (see the engine note below).

The whole command surface is two operations: VADD to insert (the graph updates live) and VSIM to query. The rest is introspection and maintenance.

§3

Similarity scores

vector-sets.md.txt · Similarity Scores

Vector Sets normalize vectors on insertion and use cosine similarity internally. Scores returned by VSIM WITHSCORES are rescaled from [-1, 1] to [0, 1] by the formula (cosine_similarity + 1) / 2:

Score	Interpretation	What it means
1.0	Identical vectors	Same direction — semantically equivalent
0.95+	Very high similarity	Semantic cache threshold for reuse
0.5	Orthogonal vectors	Unrelated in the embedding space
0.0	Opposite vectors	Maximally dissimilar

Cosine similarity measures the angle between vectors, not their magnitude — so the absolute scale of embeddings does not matter, only their direction. Most text embedding models produce direction-oriented embeddings, which makes cosine the right metric.

A score of 1.0 means identical direction; 0.5 means orthogonal (unrelated); the semantic cache threshold of 0.95 means “close enough to answer the same query without calling the LLM.”

§4

Filtered search

vector-sets.md.txt · Filtered Search

Attach JSON metadata with SETATTR on VADD, then pass FILTER 'expr' to VSIM to restrict results to elements whose metadata matches the predicate:

VADD movies VALUES 128 ... "inception" SETATTR '{"year": 2010, "genre": "scifi", "rating": 8.8}'
VSIM movies VALUES 128 ... FILTER '.year >= 2000 and .genre == "scifi"' COUNT 10

The filter grammar covers comparisons (> >= < <= == !=), logic (and or not), arithmetic (+ - * / %), and containment (value in [..]). Field selectors use dot notation (.year, .genre). Elements with missing fields or invalid JSON are silently excluded.

This is hybrid search: a vector neighbourhood AND a structured predicate in one operation. The filtered queries dive takes the grammar and FILTER-EF apart.

Filtered search combines the semantic dimension (nearest vectors) with a structured predicate (JSON metadata) in one VSIM call — the vector neighbourhood, constrained.

§5

Performance & quantization

vector-sets.md.txt · Performance Characteristics / Quantization Trade-offs

VSIM traverses the HNSW graph in O(log N) — about 50K ops/sec at 3M items and 300 dimensions. Quantization controls memory per dimension:

Type	Memory	Speed	Recall
`NOQUANT` (fp32)	4 bytes/dim	baseline	best
`Q8` (default)	1 byte/dim	~2× faster	~96%
`BIN`	1 bit/dim	~4× faster	~80%

With default Q8 quantization, a 300-dimension vector costs ~1 KB per element including graph overhead. One million items at 300 dims is roughly 1 GB. Use NOQUANT when recall loss is unacceptable; use BIN for a first-pass candidate retrieval before reranking on the full vectors.

Scale beyond a single instance by partitioning vectors across keys (shard by crc32(element) % num_shards), querying all shards in parallel, and merging by score client-side.

Q8 (default) is the practical starting point: 4× memory reduction at ~96% recall. Tune with EF on VSIM (more exploration = better recall) before reaching for NOQUANT.

Notes on ValkeyThe BCS stack runs Valkey 9, and Vector Sets are a Redis 8–only native type — Valkey core does not have VADD or VSIM. Valkey reaches vector search via the valkey-search module (C++, not core): FT.CREATE myIndex SCHEMA embedding VECTOR HNSW 6 TYPE FLOAT32 DIM 3 DISTANCE_METRIC COSINE over a Hash or JSON key, then FT.SEARCH idx "*=>[KNN 10 @embedding $vec]". Same HNSW algorithm; opposite API surface. Cite: valkey.io/topics/search · valkey.io/commands/ft.create.

§6

The grounding — standalone example, forward-tense note

vector-sets.md.txt · redis.io/docs/latest/develop/data-types/vector-sets/

redis.io/docs/latest/develop/data-types/vector-sets/ · figure 14 — the Redis 8 command surfacefrozen

VADD key VALUES 3 0.1 0.5 0.9 my-element           # add an element with an inline vector
VADD key FP32 <binary-blob> my-element             # …or a binary blob (faster for clients)
VSIM key VALUES 3 0.1 0.5 0.9 COUNT 10 WITHSCORES  # find the 10 nearest by a query vector
VSIM key ELE existing-element COUNT 10 WITHSCORES  # …or nearest to an existing element
VCARD key  VDIM key  VEMB key ele                   # count / dimension / get element's vector
VREM  key element                                   # true deletion, memory reclaimed

Vectors are not an EchoMQ or codemojex surface today. codemojex ships zero vector code — no VADD, VSIM, or embedding in echo/apps/codemojex/lib — and there is no planned spike (infra/ once held the retired cm-bitmapist bitmap spike, never a cm-*vector*). A “rooms and games you might like” recommender would be a natural fit — embed each room’s description and find similar rooms for a player via VSIM (or on Valkey, FT.SEARCH … KNN) — but this is not shipped and not specced.

the pattern

HNSW approximate-nearest-neighbour — O(log N) similarity over millions of vectors via VADD and VSIM.

its application (engine-honest + forward-tense)

The BCS stack’s Valkey reaches this via the valkey-search module (FT.CREATE … VECTOR HNSW, FT.SEARCH … KNN). A codemojex recommender — were it built — would find rooms or games by similarity, but ships none today.

A vector set answers “what is like this?” the way a sorted set answers “what is ranked near this” — a semantic neighbourhood, not an exact match. On Redis 8 that is VADD/VSIM; on the BCS stack’s Valkey it is the valkey-search module’s FT.SEARCH … KNN.

R7.04 · in this course

Bitmaps — another memory structure

R7.04 covers the bitmap pattern: a boolean per entity at 1 bit, aggregate analytics via BITCOUNT and BITOP. The modeling family: bitmap-patterns.

R7.03 · in this course

Probabilistic structures — the memory/accuracy trade

R7.03 covers HyperLogLog, Bloom, and Count-Min Sketch — structures that trade accuracy for memory. probabilistic-data-structures.

§7

The three dives

the-hnsw-graph → rag-and-recommendations → filtered-queries

Each dive takes one slice of the vector-sets pattern, in depth. Read them in order: the graph structure first, the applications second, the hybrid-search grammar third.

R7.05.1built

The HNSW graph

What a vector set is, the core commands (VADD/VSIM/VREM), the HNSW graph (O(log N) ANN, automatic), quantization (Q8/BIN/NOQUANT). The engine note lives here.

R7.05.2built

RAG and recommendations

The RAG pipeline (ingest → embed → VADD → VSIM → augment → generate), semantic caching (>0.95), recommendations (VSIM ELE), classification (kNN vote). The forward-tense codemojex note lives here.

R7.05.3built

Filtered queries

SETATTR JSON metadata + VSIM … FILTER 'expr'; the FILTER grammar (comparisons, logic, arithmetic, containment, .field selectors); FILTER-EF; hybrid search in one query.

Begin with the HNSW graph. Return to the Data Modeling & Memory chapter to continue.

§8

References

sources · related

Sources

Redis — Vector sets — the data-type overview: commands, quantization, filtered search, RAG.
Redis — VADD — add an element with its vector; quantization options.
Valkey — Search — the valkey-search module: FT.CREATE / FT.SEARCH over Hash/JSON.
Valkey — FT.CREATE — create a search index with a VECTOR HNSW schema field.
Valkey — Introducing Valkey Search — the module announcement and design rationale.

Related in this course

R7.05.1 · The HNSW graphcommands, graph structure, quantization, engine note
R7.05.2 · RAG and recommendationsRAG, semantic cache, recs, classification
R7.05.3 · Filtered querieshybrid vector + predicate search
R7.04 · Bitmapsanother structure from the modeling family
R7.03 · Probabilistic data structuresmemory/accuracy trade-offs
R7.01 · Redis as a primary databasethe chapter hub for the modeling family