Why learn Embeddings now?
You already know:
Text
↓
Tokenizer
↓
Tokens
But an application eventually needs to answer questions like:
Is "hire a vehicle"
similar in meaning to
"rent a car"?
Normal string comparison cannot reliably answer that.
Embeddings give you:
"rent a car"
↓
Embedding Model
↓
Vector A
"hire a vehicle"
↓
Embedding Model
↓
Vector B
Vector A ≈ Vector B
because their meaning is related
Official OpenAI documentation defines an embedding as a vector—a list of floating-point numbers—where vector distance can be used to estimate relatedness.
If you skip Embeddings
Later you may understand:
how to call an LLM,
how prompts work,
how agents call tools,
but struggle to understand:
Documents
↓
Chunks
↓
Embeddings
↓
Vector Database
↓
Similarity Search
↓
Relevant Chunks
↓
RAG
↓
Agent
That exact embedding → vector DB → RAG progression appears later in your roadmap.
Concepts Covered
Learning Tree
Embeddings
│
├── Must Learn Now
│ ├── What an embedding is
│ ├── Text → vector
│ ├── Vector / dimensions
│ ├── Semantic similarity
│ ├── Similar meaning → nearby vectors
│ ├── Basic cosine similarity
│ ├── Token embedding
│ ├── Application-level embedding
│ └── Why RAG needs embeddings
│
├── Learn at High Level
│ ├── Vector space
│ ├── Distance / similarity
│ ├── Nearest-neighbour search
│ ├── Query embedding
│ └── Document/chunk embedding
│
├── Learn Later
│ ├── Vector databases
│ ├── pgvector
│ ├── Chunking strategies
│ ├── Top-K retrieval
│ ├── Metadata filtering
│ ├── Hybrid search
│ ├── Reranking
│ └── Retrieval evaluation
│
└── Do Not Learn Yet
├── Training embedding models
├── Contrastive-learning mathematics
├── Backpropagation derivations
├── GPU implementation
├── ANN algorithm internals
└── Research papers on embedding geometry
The roadmap explicitly limits Step 4 to the embedding definition, text-to-vector idea, semantic similarity, nearby vectors, token-vs-application embeddings, RAG relevance, and only a very basic cosine-similarity concept. “Embeddings deeper” is postponed to Project 3.
What Theory Should You Learn?
Use your requested balance:
Theory ≈ 30%
Practical Experiments ≈ 50%
Debug / Compare ≈ 20%
3.1 What is an Embedding?
Definition:
An embedding is a numerical vector representing information in a way that allows software to compare meaning or relatedness.
Official OpenAI documentation describes embeddings as floating-point vectors whose distance indicates relatedness.
Why it exists:
Computers cannot directly perform semantic mathematical comparisons on sentences.
Before embeddings:
Query: "hire a vehicle"
Database:
"rent a car"
Exact keyword comparison may miss the relationship.
With embeddings:
hire a vehicle
↓
vector
↓
compare
↓
rent a car
Why Agentic AI engineers care:
An agent often needs to retrieve relevant knowledge before reasoning.
Embeddings in Increasing Depth
Level 1 — One-Line Explanation
Embedding converts meaning into numbers so software can compare similar information.
Level 2 — Beginner Explanation
Imagine every sentence receives a location on a huge meaning map.
Vehicle concepts
"rent a bike"
●
● "hire a motorcycle"
● "rent a car"
● "vehicle rental"
Food concepts
● "order pizza"
● "buy dinner"
Related meanings tend to occupy related regions.
That conceptual idea is what semantic embedding search exploits.
Level 3 — Developer Explanation
As a developer, think of an embedding service almost like a transformation API:
String
↓
Embedding model
↓
float[]
Example conceptual output:
"cardiology hospital"
→
[
0.128,
-0.374,
0.821,
...
]
You normally do not interpret individual numbers.
Instead you compare the complete vectors.
query_vector
↓
similarity(query_vector, document_vector)
↓
score
Level 4 — Agentic AI Engineer Explanation
LLM Application
↓
RAG
↓
Retrieval
↓
Embeddings
↓
Vector Search
↓
Relevant Context
↓
LLM
↓
AI Agent
↓
Tools + State + Workflow
↓
Production Agentic AI
Embeddings are not the agent.
They solve a narrower problem:
Find information that is semantically related to this query.
The roadmap places embeddings under the RAG side of the eventual Agentic AI architecture rather than treating them as reasoning or orchestration themselves.
Why Were Embeddings Needed?
Consider this search.
Data
Hospital A:
"Advanced cardiac treatment available"
Hospital B:
"Orthopedic knee replacement centre"
User
"I need treatment for my heart."
A simple keyword system searches:
heart == cardiac ?
No.
A semantic embedding system instead compares meaning.
"heart treatment"
↓
Embedding
↓
Semantic similarity
↓
"cardiac treatment"
↓
Strong relationship
So:
Older Approach
Keyword matching
↓
Problem
Different words can express similar meaning
↓
Embeddings
Represent meaning numerically
↓
Improvement
Semantic matching becomes possible
Embeddings are widely used for similarity/search-style tasks because semantically related inputs produce related vector representations.
Important:
Embeddings do not make keyword search obsolete.
Exact identifiers such as:
Booking ID: BK-98271
Phone: 9876543210
Hospital ID: 128
are often better handled with exact/database filters.
Hybrid search comes later.
Complete Internal Working Flow
Input Text
"Find a heart specialist"
│
│ text
↓
┌─────────────────────┐
│ Embedding Model │
└──────────┬──────────┘
│ numeric vector
↓
[0.18, -0.42, 0.71, ...]
│
│ compare against stored vectors
↓
┌─────────────────────┐
│ Similarity Function │
└──────────┬──────────┘
│ similarity scores
↓
Candidate documents ranked
│
↓
"Cardiology specialists..."
Step 1 — Input
Normal application text.
"Find a heart specialist"
Step 2 — Embedding Model
The model maps text into a vector representation.
Step 3 — Vector
Conceptually:
[0.18, -0.42, 0.71, ...]
Each number is one coordinate of the representation.
Step 4 — Comparison
Compare the query vector against candidate vectors.
Step 5 — Ranking
cardiologist → high similarity
heart doctor → high similarity
orthopedic surgeon → lower
hotel reservation → much lower
Step 6 — Use Result
Later in RAG:
Top relevant chunks
↓
LLM context
Important Components
Simple Real-World Example
Vehicle Rental Search
Suppose Moto rental descriptions contain:
A: "Book a rental car"
B: "Hire a vehicle"
C: "Reserve a hotel room"
D: "Rent a motorcycle"
User enters:
"I need a car on rent."
Flow:
User Query
"I need a car on rent"
↓
Embedding
↓
Query Vector
↓
Compare with stored vectors
↓
Similarity ranking
↓
A — Book a rental car
B — Hire a vehicle
D — Rent a motorcycle
C — Reserve a hotel room
The roadmap itself recommends a small Project-0 experiment based on semantically comparing rental-related sentences instead of building a complete application.
Agentic AI Example
Imagine:
User:
"Find hospitals experienced in knee replacement
and then check appointment availability."
Architecture:
User Request
↓
AI Agent
↓
Agent decides:
"I need hospital knowledge"
↓
Retrieval Tool
↓
Query Embedding ← EMBEDDINGS
↓
Vector Search
↓
Relevant Hospital Profiles
↓
Agent Reasons
↓
Appointment API
↓
Available Slots
↓
LLM
↓
Final Answer
This part IS handled by embeddings
"knee replacement specialist"
↓
vector
↓
find semantically relevant hospital information
This part is NOT handled by embeddings
Embeddings do not:
decide which tool to call
plan tasks
call APIs
book appointments
maintain workflow state
approve actions
generate the final answer
Those belong to LLM/agent/tool/workflow components later in your roadmap.
Embeddings vs Closely Related Concepts
10.1 Embedding vs Token
Token Embedding vs Application-Level Embedding
This distinction is extremely important.
Token embedding
Inside an LLM:
Token
↓
Token ID
↓
Token Embedding
↓
Transformer
Example:
"doctor"
↓
token embedding
This is an internal representation used during model processing.
Application-level embedding
Used by your application:
"Hospital provides cardiac surgery"
↓
Embedding API/model
↓
[0.12, -0.48, ...]
↓
Vector Search
Use cases include semantic search and retrieval.
Therefore:
Token embedding
≠
RAG/document embedding
10.3 Embedding vs Vector Database
Embedding Model
↓
Vector
↓
Vector Database
Embeddings
↓
Vector DB
↓
RAG
Embeddings vs RAG
Embeddings ≠ RAG
RAG is an entire pipeline.
Documents
↓
Chunking
↓
Embeddings
↓
Vector storage
↓
Retrieval
↓
Relevant context
↓
LLM
↓
Answer
Embeddings are one component of many RAG architectures. Your roadmap explicitly shows this sequence
Embeddings vs Generation
Embedding model
Text → Vector
Generative LLM
Text/context → Generated tokens
Embedding output:
[0.17, -0.38, ...]
LLM output:
"Here are three hospitals..."
Do not confuse them.
Relationship with Previous and Next Concepts
T
okens / Tokenization
↓
Creates model-readable units
↓
Embeddings
↓
Creates numerical representations
↓
Later: Vector Database
↓
Semantic retrieval
↓
RAG
What should you already know?
You should understand:
token,
tokenization,
token vs word,
high-level Transformer flow,
high-level attention/context.
These are the exact Project-0 concepts preceding Embeddings in the roadmap.
What becomes easier after Embeddings?
You will understand why:
"heart doctor"
can retrieve:
"cardiology specialist"
without exact keyword equality.
Future concepts depending on this
Vector DB
Semantic Search
RAG
Advanced RAG
Agentic RAG
Long-term knowledge retrieval
Retrieval-enabled agents
What You Should NOT Learn Now
This is important.
❌ Training an embedding model from scratch
Reason: Not needed to build production agent applications.
Learn later when: You specifically move into ML/model engineering.
❌ Backpropagation equations
Reason: Does not improve your current ability to use/debug embeddings.
Learn later when: Studying model training.
❌ Detailed linear-algebra proofs
Reason: Basic vector intuition is sufficient now.
Learn later when: Your work requires research-level optimization.
❌ PyTorch implementation of embedding layers
Reason: You need application behaviour, not model implementation.
Learn later when: Building/customizing neural models.
❌ HNSW/IVF/PQ internals
These are vector-indexing topics.
Reason: They belong to deeper vector-database engineering.
Learn later when: Project 3 or production scale requires retrieval-performance tuning.
❌ Advanced embedding fine-tuning
Reason: Premature.
Learn later when: Generic embeddings fail significantly on specialized domain retrieval.
❌ Deep cosine-similarity mathematics
Know only:
Two vectors
↓
Compare orientation
↓
Get similarity score
Basic formula awareness is enough:
cosine_similarity(A,B)
=
(A · B) / (||A|| × ||B||)
The roadmap specifically says very basic cosine similarity, and separately warns against spending early learning time on advanced mathematics and architecture research.
Practical Experiments
Experiment 1 — Semantic Similarity Explorer
Goal
See whether similar sentences receive similar embeddings.
Input
"Book a rental car"
"Hire a vehicle"
"Reserve a hotel"
"Rent a motorcycle"
What to do
Generate embeddings.
Select "Book a rental car" as the query.
Calculate similarity against the others.
Sort highest → lowest.
Expected observation
Vehicle-rental sentences should generally rank closer than unrelated hotel-booking text.
Concept proved
Meaning
↓
Embedding
↓
Similarity
Common mistake
Expecting identical wording to be required.
This is also the style of embedding experiment explicitly recommended for Project 0.
Experiment 2 — Synonym Search
Input
Query:
"heart doctor"
Documents:
"cardiology specialist"
"skin specialist"
"orthopedic surgeon"
"cardiac care physician"
Observe
Semantic search should identify the cardiology/cardiac descriptions as more related than unrelated specialties.
Concept proved
word equality
≠
semantic similarity
Experiment 3 — Meaning Changes With Context
Input
"Java developer"
"Java programming language"
"Java island tourism"
"coffee from Java"
Generate embeddings and compare them.
Observe
The same text fragment "Java" does not force every sentence to occupy the same semantic region.
Concept proved
Embeddings represent the input as a whole, not merely isolated matching words.
Debugging / Failure Experiment
Embeddings are useful, but not magic.
Normal case
Query:
"heart specialist"
Candidate:
"cardiology doctor"
↓
Expected:
Strong semantic relationship
Ambiguous case
Query:
"bank services"
Candidate A:
"loan and savings account"
Candidate B:
"activities near the river bank"
Depending on surrounding text and embedding model behaviour, ambiguous/underspecified queries can produce imperfect rankings.
Ambiguous query
↓
Weak semantic context
↓
Embedding
↓
Several plausible neighbours
↓
Unexpected retrieval ranking
Why?
The embedding has only the information contained in the input.
Bad:
"bank"
Better:
"bank account and personal loan services"
Lesson
When retrieval is poor, do not immediately conclude:
"Vector database is broken."
Investigate:
Query quality
↓
Chunk quality
↓
Embedding model
↓
Similarity/search configuration
↓
Metadata/filtering
↓
Ranking
The latter stages belong mainly to Project 3.
Small Python Experiment
For learning purposes, the current OpenAI embedding API accepts text and returns embedding vectors that can then be stored or compared.
from openai import OpenAI
import numpy as np
client = OpenAI()
texts = [
"Book a rental car",
"Hire a vehicle",
"Reserve a hotel room",
"Rent a motorcycle",
]
response = client.embeddings.create(
model="text-embedding-3-small",
input=texts
)
vectors = [
np.array(item.embedding)
for item in response.data
]
def cosine_similarity(a, b):
return np.dot(a, b) / (
np.linalg.norm(a) * np.linalg.norm(b)
)
query = vectors[0]
for text, vector in zip(texts[1:], vectors[1:]):
score = cosine_similarity(query, vector)
print(text, round(score, 4))
The model name shown here is documented in OpenAI's current Embeddings guide.
Flow
Input strings
↓
Embedding API
↓
Vectors
↓
Cosine similarity
↓
Similarity scores
↓
Compare ranking
What you learned
Not:
How to build an embedding neural network
But:
How an Agentic AI application actually USES embeddings
That is the correct depth for Project 0.
- Architecture Diagrams 16.1 Diagram A — End-to-End Pipeline ┌─────────────────────────┐ │ Input Text │ │ "Find a heart doctor" │ └────────────┬────────────┘ │ text ↓ ┌─────────────────────────┐ │ Embedding Model │ └────────────┬────────────┘ │ numerical representation ↓ ┌─────────────────────────┐ │ Embedding Vector │ │ [0.18,-0.42,0.71,...] │ └────────────┬────────────┘ │ compare with candidates ↓ ┌─────────────────────────┐ │ Similarity Calculation │ └────────────┬────────────┘ │ ranked scores ↓ ┌─────────────────────────┐ │ Relevant Text │ │ "Cardiology doctor..." │ └─────────────────────────┘ 16.2 Diagram B — Internal Anatomy
At your current depth:
┌──────────────────── EMBEDDING PROCESS ────────────────────┐
│ │
│ Text │
│ │ │
│ │ tokenize/process │
│ ↓ │
│ Model Input │
│ │ │
│ │ encode │
│ ↓ │
│ Learned Representation │
│ │ │
│ │ produce fixed numeric representation │
│ ↓ │
│ Embedding Vector │
│ │
└───────────────────────────────────────────────────────────┘
There is no application-level “repeat until done” loop inside normal embedding generation that you need to understand at Project-0 depth.
Later retrieval repeats vector comparisons across candidates/index structures, but that belongs to vector-search engineering.
16.3 Diagram C — Ecosystem Hierarchy
Your prompt suggests:
Embedding ⊂ ... ⊂ LLM ⊂ LLM Application ⊂ Agent...
That needs one correction.
There are two different embedding contexts.
Internal token embeddings
Token
↓
Token Embedding
↓
Transformer
↓
LLM
↓
LLM Application
Application-level embeddings
Embedding Model
↓
Embedding Vector
↓
Vector Search / Retrieval
↓
RAG Component
↓
LLM Application
↓
AI Agent
↓
Agentic AI System
So it would be misleading to say that every application-level embedding is literally “inside the LLM.”
What outer layers add
Embedding
→ semantic representation
Retrieval
→ finds relevant information
RAG
→ supplies retrieved information to an LLM
LLM Application
→ combines model + application logic
AI Agent
→ adds decisions + tools + actions
Agentic AI System
→ adds workflows, state, memory,
orchestration, governance, multiple actions/agents
Top comments (0)