A practical walkthrough of embedding models, vector stores, retrieval strategies, prompt engineering, and evaluation — without leaving the JVM.
1. Why RAG — and Why Java Developers Need It
Large language models are impressive, but they have a fundamental limitation: their knowledge is frozen at training time. Ask a model about your internal product documentation, your company’s latest policy changes, or a codebase that postdates its training cutoff, and you will get either a hallucination or a polite refusal. Retrieval-Augmented Generation (RAG) fixes exactly this problem.
The idea is straightforward: instead of relying solely on the model’s baked-in knowledge, you first retrieve relevant documents from your own data store and then pass those documents as context alongside the user’s question. The model’s job shifts from “remember the answer” to “read these passages and synthesise a response.” That shift makes answers both more accurate and more explainable.
For Java teams, however, most RAG tutorials live in the Python ecosystem — LangChain, LlamaIndex, and friends. Fortunately, LangChain4j brings the same pipeline primitives to the JVM, and it integrates cleanly with Spring Boot, Quarkus, or plain Maven projects. You do not need to introduce a Python service, a foreign runtime, or a polyglot deployment just to add intelligent search to your Java application.
Who is this for?Java developers who want to add AI-powered document search, customer-support bots, or internal knowledge bases to existing JVM applications — without abandoning the tools and deployment pipelines they already know.
2. The Five Stages of a RAG Pipeline
Before writing a single line of code, it helps to see the whole pipeline at once. Every production RAG system — regardless of language or framework — passes through the same five stages.
Think of it as a librarian who, rather than guessing what a book says, first walks to the relevant shelf, pulls out the right pages, and only then answers your question. Each stage has meaningful configuration choices that directly affect answer quality — and we will cover the important ones below.
RAG vs Plain LLM — Answer Accuracy on Domain-Specific Queries
3. Setting Up LangChain4j
LangChain4j follows a modular design. You add only the components you need, so your build stays lean. The core dependency brings the abstractions; separate artifacts add specific model providers or vector stores.
Add the following to your pom.xml. This combination gives you the OpenAI embedding model, an in-memory vector store suitable for development, and the full RAG pipeline utilities:
<!-- Core LangChain4j RAG abstractions --> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j</artifactId> <version>0.35.0</version> </dependency> <!-- OpenAI integration (embeddings + chat) --> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j-open-ai</artifactId> <version>0.35.0</version> </dependency> <!-- In-memory embedding store (swap for Chroma/Qdrant in production) --> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j-embeddings-all-minilm-l6-v2</artifactId> <version>0.35.0</version> </dependency>
Tip — local embeddings for developmentThe
langchain4j-embeddings-all-minilm-l6-v2artifact bundles the all-MiniLM-L6-v2 model as a JAR dependency via Deep Java Library (DJL). It runs entirely on the JVM — no Python, no separate inference server — which is perfect for local development and CI pipelines.
4. Embedding Your Documents
Embedding is the process of turning text into a fixed-length numeric vector that captures its semantic meaning. Two sentences that mean the same thing should produce vectors that are geometrically close; unrelated sentences should be far apart. That geometric relationship is precisely what powers semantic search.
LangChain4j exposes a clean EmbeddingModel interface, which means you can swap providers — OpenAI, Cohere, a local Ollama model, or the bundled MiniLM — without changing the rest of your pipeline.
4.1 Loading and Chunking Documents
Chunking strategy is one of the most impactful tuning levers in any RAG pipeline. Chunks that are too large give the retriever too much noise to wade through; chunks that are too small lose the context that makes a passage meaningful. A window of 300–500 tokens with 20% overlap is a solid starting point for most prose documents.
// 1. Load a plain text or PDF document
Document document = FileSystemDocumentLoader.loadDocument(
Path.of("docs/product-manual.txt"),
new TextDocumentParser()
);
// 2. Split into overlapping chunks (500 tokens, 50-token overlap)
DocumentSplitter splitter = DocumentSplitters.recursive(
500, // max chunk size in characters
50 // overlap between adjacent chunks
);
List<TextSegment> segments = splitter.split(document);
// 3. Choose an embedding model (local, no API key required)
EmbeddingModel embeddingModel = new AllMiniLmL6V2EmbeddingModel();
// 4. Build an in-memory store and ingest all segments
EmbeddingStore<TextSegment> store =
new InMemoryEmbeddingStore<>();
EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
.embeddingModel(embeddingModel)
.embeddingStore(store)
.build();
ingestor.ingest(segments);
That is all it takes to move from a raw file to a queryable semantic store. In production, you would swap InMemoryEmbeddingStore for a persistent option such as Qdrant, Chroma, or Weaviate — each of which LangChain4j supports via a matching integration artifact.
| Vector Store | LangChain4j Artifact | Best For | Persistence |
|---|---|---|---|
| InMemory | langchain4j (core) | Development, unit tests | None (RAM only) |
| Chroma | langchain4j-chroma | Local development, prototyping | Docker container |
| Qdrant | langchain4j-qdrant | Production, high throughput | Cloud or self-hosted |
| PGVector | langchain4j-pgvector | Existing PostgreSQL stack | PostgreSQL |
| Weaviate | langchain4j-weaviate | Hybrid search (vector + BM25) | Cloud or self-hosted |
5. Retrieval: Finding the Right Chunks
Retrieval is the heart of any RAG system. When a user submits a question, you embed that question with the same model used during ingestion, then find the k stored chunks whose vectors are most similar. LangChain4j makes this a one-liner with EmbeddingStoreRetriever.
// Wrap the store in a content retriever
ContentRetriever retriever = EmbeddingStoreContentRetriever.builder()
.embeddingStore(store)
.embeddingModel(embeddingModel)
.maxResults(5) // return top-5 chunks
.minScore(0.6) // discard low-confidence matches
.build();
// Wire everything into an AI service
interface ProductAssistant {
String answer(String question);
}
ProductAssistant assistant = AiServices.builder(ProductAssistant.class)
.chatLanguageModel(OpenAiChatModel.withApiKey("YOUR_API_KEY"))
.contentRetriever(retriever)
.build();
// That's it — the pipeline runs end-to-end on every call
String answer = assistant.answer(
"What is the return policy for defective items?"
);
5.1 Improving Results with a Reranker
Cosine similarity is fast and generally effective, but it can occasionally surface chunks that are semantically adjacent to the query without being genuinely relevant. A reranker — a second, more expensive model — re-scores the top-k candidates for actual relevance to the specific question. As the chart above shows, adding a reranker typically pushes accuracy up by 10–15 percentage points. LangChain4j supports Cohere Rerank as a drop-in ContentRetriever decorator.
Chunk Size vs. Retrieval Precision
6. Prompt Engineering for RAG
Retrieval brings the right passages into the conversation; prompt engineering tells the model what to do with them. A well-structured prompt significantly reduces hallucinations, keeps answers grounded in the retrieved text, and helps the model signal uncertainty when the context does not contain the answer.
LangChain4j lets you define prompt templates using its @SystemMessage and @UserMessage annotations on your AI service interface. The framework automatically injects the retrieved context into the {{context}} placeholder at runtime.
interface GroundedAssistant {
@SystemMessage("""
You are a helpful assistant for ACME Corp customers.
Answer ONLY using the information in the context below.
If the context does not contain the answer, say:
"I don't have that information in the documentation."
Context:
{{context}}
""")
String answer(@UserMessage String question);
}
Notice the explicit instruction to admit when the answer is not in the context. Without this guardrail, language models will confidently fabricate an answer — the very behaviour RAG is meant to prevent. This single line of prompt engineering often makes the biggest practical difference in production.
Common pitfall — context window overflowEach retrieved chunk consumes tokens. If you retrieve 10 large chunks for every query, you may exceed the model’s context window or incur unexpected costs. Start with 3–5 chunks, measure faithfulness scores (see the next section), and increase only if needed.
6.1 Query Rewriting
Short or ambiguous user questions sometimes miss relevant chunks because the vocabulary does not match the stored documents. A simple but effective technique is query rewriting: use a fast LLM call to expand the user’s question into a richer search query before hitting the vector store. LangChain4j’s QueryTransformer interface supports this pattern out of the box, and it requires very little additional cost per request.
7. Evaluating Your Pipeline
A RAG pipeline that you cannot measure is a RAG pipeline you cannot improve. Fortunately, the field has converged on three straightforward metrics that together give you a complete picture of pipeline health. These can be computed programmatically, which means you can include them in your CI pipeline and catch regressions before they reach production.
| Metric | What It Measures | Ideal Value | Tool |
|---|---|---|---|
| Faithfulness | Does the answer rely only on the retrieved context? | > 0.85 | RAGAS, custom LLM judge |
| Answer Relevancy | Does the answer actually address the question? | > 0.80 | RAGAS, cosine similarity |
| Context Precision | Were the retrieved chunks actually useful? | > 0.75 | RAGAS, manual annotation |
The most practical evaluation approach for a Java team is to build a small golden dataset — 50 to 100 representative question-answer pairs drawn from your actual domain — and run each pair through the pipeline on every CI build. You can use LangChain4j’s chat model interface to implement a lightweight LLM-as-judge: ask the model to rate faithfulness on a 1–5 scale and flag any response that drops below your threshold.
Evaluation Score Breakdown — Example Production Pipeline
7.1 A Minimal LLM-as-Judge in Java
You do not need a separate evaluation framework to get started. The following pattern uses LangChain4j itself to evaluate its own outputs, which means you already have everything you need once the main pipeline is wired up.
interface FaithfulnessJudge {
@SystemMessage("""
You are an evaluation assistant. Given a CONTEXT and an ANSWER,
rate how faithfully the answer is grounded in the context.
Reply with ONLY a JSON object: {"score": <1-5>, "reason": "..."}
""")
String evaluate(
@UserMessage String contextAndAnswer
);
}
// Usage — call this in your test suite for each golden pair
String prompt = String.format(
"CONTEXT:\n%s\n\nANSWER:\n%s",
retrievedContext,
generatedAnswer
);
String result = judge.evaluate(prompt);
// Parse result JSON and assert score >= 4
This kind of automated evaluation loop, combined with a thoughtfully chosen golden dataset, closes the feedback cycle between deployment and iteration — which is ultimately what separates a production pipeline from a proof of concept.
For more advanced evaluation, RAGAS is the de-facto open-source toolkit. While its native API is Python, you can run it as a sidecar evaluation service and call it over HTTP from your Java test suite. Alternatively, the LangChain4j documentation has a growing section on evaluation utilities being added natively to the library.
8. What We Have Learned
In this article, we walked through the complete journey of building a production-ready RAG pipeline on the JVM. Specifically, we covered:
- Why RAG matters for Java teams — it adds live, domain-specific knowledge to any LLM without leaving the JVM or retraining a model.
- The five-stage pipeline — ingestion, chunking, embedding, retrieval, and generation — and how each stage interacts with the next.
- LangChain4j’s modular Maven setup — core, OpenAI integration, local MiniLM embeddings, and the in-memory store for development.
- Chunking strategy — 300–500 characters with overlap is a strong starting point; the chunk-size benchmark confirms this is where precision peaks.
- Retrieval tuning — cosine similarity with a
minScorethreshold, plus optional reranking for higher-stakes use cases. - Prompt engineering guardrails — explicitly instructing the model to cite only the context and admit uncertainty when the answer is absent.
- Evaluation fundamentals — faithfulness, answer relevancy, and context precision, measured against a golden dataset in your CI pipeline.
