How Embeddings Work

From the ground up. Connected to what you're building.

What embedding actually does

You give it text — a word, sentence, paragraph, whatever. It returns a list of numbers. Your model (Qwen3-Embedding-0.6B) returns exactly 1,024 numbers. That's it. That list of 1,024 floats IS the embedding.

"Thompson Sampling" → [0.0234, -0.1891, 0.0012, ..., 0.0847]  (1,024 values)
"Banana split"      → [0.1102, 0.0034, -0.2201, ..., -0.0391]  (1,024 values)

Each of those 1,024 positions is a dimension. Not a "chunk" of the text — a dimension of meaning. The model learned during training that, say, dimension 47 correlates with "technical vs casual," dimension 312 correlates with "concrete vs abstract," dimension 891 correlates with "positive vs negative." But you don't get to pick what each dimension means — the model discovered them during training. Most dimensions don't map to anything a human would name.

How does the model decide what numbers to put there?

The text goes through the transformer layers — attention heads, feedforward networks, the whole stack. The final hidden state (the last layer's output) gets pooled (usually averaged across all token positions) into a single 1,024-dim vector. That vector is the embedding.

The model was trained with a contrastive objective: "these two texts are similar, push their vectors close together. These two texts are different, push them apart." After millions of such comparisons, the 1,024 dimensions organize themselves into a space where cosine similarity between vectors ≈ semantic similarity between texts.

What you're actually measuring when you do cosine similarity

similarity = dot(embedding_A, embedding_B) / (norm(A) * norm(B))

This is the angle between two 1,024-dimensional arrows. Close to 1.0 = pointing the same direction = similar meaning. Close to 0 = perpendicular = unrelated. Close to -1.0 = opposite.

The key thing you're asking about — what gets embedded?

This is the "defactorization" question. You choose what text to feed in:

What your system currently does

Each of your 205 skills gets embedded as one vector (the full skill description, maybe 50-100 words → 1,024 dims). Each task gets embedded similarly. You compare them with cosine similarity. That's your relevance dimension.

What "different factorization protocols" could mean

1. Chunk size: Embed skills at sentence level instead of full description. More vectors per skill, finer-grained matching, but more computation.
2. Multi-vector: Instead of one 1,024-dim vector per skill, keep multiple vectors (one per sentence). ColBERT does this — it compares every query token against every document token and takes the best matches. Much more expensive, much more precise.
3. Dimensionality reduction: Take the 1,024 dims and project down to, say, 128 via PCA or learned projection. Faster, loses some signal. Matryoshka embeddings (which Qwen3 supports) let you truncate to any prefix length.
4. Factored embeddings: Separate the embedding into interpretable sub-spaces — dimensions 0-255 for "topic," 256-511 for "style," 512-767 for "complexity," 768-1023 for "specificity." Nobody's done this well because the dimensions aren't naturally organized that way. But if you COULD... that's where your R/I/V weighting would operate at the embedding level, not just the retrieval scoring level.