Link to chatbot: https://dale-carnegie-ai.vercel.app/
I recently built a chatbot based on Dale Carnegie’s book - How to Win Friends and Influence People as part of a team with two other amazing developer(Fayez & Elberd). To make it effective, we used Open AI embeddings and a method called RAG (Retrieval-Augmented Generation). In this post, I’ll explain what embeddings are, how we applied them to create a RAG chatbot, and whether they’re useful beyond that. Let’s dive in.
What Are Open AI Embeddings?
Open AI embeddings are a way to turn text into a numerical representation(basically numbers). When you give the Open AI embedding tool a piece of text—like a sentence or paragraph—it outputs a list of 1,536 numbers. These numbers capture the meaning of the text, not just the words themselves.
For example, “Listen carefully” and “Pay attention to others” might have similar numerical representations because they express related ideas.
The process uses Open AI’s language models, trained on massive amounts of text, to understand context and relationships between words. The result is a set of numbers that represents the text’s meaning in a way computers can process.
How We Used Embeddings for our RAG Chatbot
Our goal was to build a chatbot that answers questions using only Dale Carnegie’s book. RAG helped us do that by combining retrieval and generation. Here’s how embeddings made it work:
Splitting the Book: We sourced the PDF version of ‘How to win friends and influence people’. Used a parsing library - PDF.js to extract text and split it into smaller pieces, like paragraphs or sections using some custom code. Each piece was sent to Open AI’s embedding tool, which turned it into a list of 1,536 numbers.
Storing the Numbers: These numerical representations were saved in a vector database (Pinecone), creating a searchable collection of the book’s content organised by meaning.
Handling Questions: When a user asked, “How do I connect with people?” the query was converted into a numerical representation by the embedding tool. The code then compared it to the stored book embeddings in the database to find the closest matches, like “Show genuine interest in others.”
Creating Answers: The matching text pieces were sent to a language model, which rewrote them into a clear, conversational response based on Carnegie’s words.
Embeddings were essential because they allowed the code to search the book by meaning, not just exact word matches. This made the chatbot more flexible and accurate.
How to Use Embeddings for Your Own RAG Project
If you want to build something similar, here’s the basic process:
Choose Your Text: Pick a document or set of documents you want your chatbot to use.
Generate Embeddings: Use Open AI’s API to convert your text into numerical representations. You’ll need some coding knowledge.
Store the Data: Save the numbers in a vector database like Pinecone.
Add a Model: Connect it to a language model (OpenAI or Claude) to turn retrieved text into answers.
This setup works well for chatbots that need to stick to specific information, like support bots or personal knowledge tools.
Are Embeddings Only for RAG?
No, RAG is just one application. Open AI embeddings have other uses because they’re great at capturing text meaning in numbers. Here are some examples:
Text Similarity: Compare two numerical representations to see how close their meanings are. This can help with tasks like finding duplicate content or matching documents.
Clustering: Group similar text based on their numbers. For instance, sorting feedback into categories like “positive” or “negative” without reading everything.
Search Tools: Create a search system that finds text by meaning, not just keywords. A query like “ways to stay focused” could return “tips for concentration.”
Sentiment Detection: Analyse whether text is positive, negative, or neutral by comparing its numerical representation to known examples.
For our project, we focused on RAG to keep the chatbot tied to Carnegie’s book, but embeddings can support many other ideas if you experiment.
What do you think? Have any ideas for using embeddings? I’m planning to build on this concept and bring other books to life as chatbots I can talk to! Let me know your thoughts in the comments!