Embedding Lab

water pulls up things like drinking, salt, fresh, supply, gas, air — not just synonyms, but the whole situation water appears in. three sits with other numbers, red with other colours, london with other cities.

Nobody told the model that "three" is a number. It read a lot of text, noticed that three and four get used in near-identical sentences, and placed them next to each other. Category structure fell out of raw co-occurrence.

The distance number matters. That's cosine distance — smaller means more similar. This is the actual quantity production systems use for search and recommendations, computed on all 200 dimensions. Not on the picture.

Antonyms are extremely close — often in each other's top 10. Most people predict the opposite, and being wrong here is the most useful thing in this lab.

Why: embeddings learn from company kept. Think about the sentences these words appear in: "the food was good" / "the food was bad." "turn the heat up" / "turn the heat down." Antonyms are grammatically interchangeable and appear in near-identical contexts, so distributional learning puts them almost on top of each other.

So the axis embeddings capture is relatedness — same topic, same slot in a sentence — not sameness of meaning and definitely not sentiment. This is a real, load-bearing limitation: naive embedding search for "great product" will happily surface "terrible product."

One blended list. bank typically leans heavily financial (that sense dominates the training corpus) with a few geographic words mixed in. apple mixes fruit and company. rock mixes stone and music.

There is exactly one row in the table for "bank." The model cannot give you two — so it stores a single compromise vector, roughly a frequency-weighted average of every sense, sitting in a slightly incoherent place that belongs fully to neither meaning.

Notice the rarer sense often gets crushed. If 90% of "bank" occurrences are financial, the vector parks itself in finance-land and river-bank is a faint pull.

Three methods, three different pictures — and t-SNE gives you a different layout every time you run it, because it starts from a random initialisation. The vectors never changed. Only the shadow did.

The PCA variance number is the honest one to stare at: squashing 200 dimensions into 3 typically retains only tens of a percent of the variance. You are looking at a small fraction of the information and calling it the data.

The critical trap: in t-SNE and UMAP, distance between clusters is not meaningful. Both are designed to preserve local neighbourhoods and will happily distort global distances to make a readable picture. Two clusters drawn far apart may not be far apart. Cluster sizes and empty space between blobs are largely artefacts.

The pairs line up. king/uncle/actor/father sit toward the "man" end; queen/aunt/actress/mother sit toward the "woman" end — and each pair is separated by roughly the same displacement.

That's the key insight: the space encodes concepts not only as positions but as directions. There is a consistent "gender" direction, and adding it to king lands you near queen. That's literally what king − man + woman ≈ queen is doing — subtract to isolate the direction, add it somewhere else.

The same holds for size, time, and price axes. A single direction, reusable across many word pairs.

A clear, uncomfortable split. engineer, doctor, scientist, captain lean toward "man"; nurse, secretary, teacher, dancer lean toward "woman." You didn't tell the model anything about gender roles — you built an axis from two words and the occupations sorted themselves.

Where it comes from: the corpus. The model read enormous amounts of human-written text in which nurses are described as "she" and engineers as "he." It has no beliefs; it's a statistical mirror of what people wrote. The bias is a faithful measurement of the training data, which is exactly what makes it dangerous — it's real, learned, and invisible until you go looking.

Why it matters in production: put this embedding inside a résumé search or a job-ad targeter, and it will rank a woman's engineering CV lower for reasons no one coded and no one can see in the logs. Bias here isn't abstract — it propagates into every downstream system that consumes the vectors.

Mostly the same words, somewhat reordered — and after you tick Sphereize, the two metrics agree far more closely.

Why: cosine measures the angle between vectors and ignores their length. Euclidean measures straight-line distance, so length counts. In word embeddings, vector length correlates with things like word frequency — not meaning. Using Euclidean lets "how common is this word" leak into "how similar is this word."

"Sphereize" normalises every vector to unit length. Once all vectors sit on the same sphere, ranking by cosine and ranking by Euclidean become equivalent — which is exactly why production systems normalise their embeddings and then use a plain dot product.

Ten reasonably clean clusters — with predictable trouble spots: 4 and 9 bleed together, as do 3, 5 and 8, and 1 and 7. Click into the overlaps and you'll find genuinely ambiguous handwriting.

No words anywhere. Same trick: pixels in, vectors out, and similar things land near each other. The geometry now encodes visual similarity instead of semantic similarity.

The useful part: where clusters touch is where a classifier makes its mistakes. The confusion matrix you'd get from a trained model is visible in the geometry before you train anything. This is a real debugging technique — embed your data, look at what overlaps, and you've found your hard cases.