RAG Knowledge Base

Problem

Finding the Same Evidence Again to Understand Each Feature

Requirements and discussions lived in Jira, background and policy in Confluence, and implementation and change history in Git. Finding material in each tool still left someone to check whether decisions made in a ticket had reached the policy and code. AI development tools also required repeatedly finding sources, copying them, and explaining the context again.

QA required the same investigation. E2E reduced repetitive checks of basic features, but new and changed features still needed requirements analysis, impact assessment, and test-case design. Accounting for past bugs and exceptions in related features meant returning to several tools, with the scope affected by what the reviewer remembered.

I built an internal RAG platform to retrieve and compare this distributed knowledge together. I applied it first to QA preparation and verification, then extended it to development assistance and policy review during planning.

Retrieval Design

From Similar Topics to the Conditions a Question Needs

I first collected and indexed material from Jira, Confluence, and Git alongside QA records. I started with dense retrieval, embedding questions and documents to find passages with similar meaning. The aim was to retrieve material without knowing its exact terminology, such as finding the temporary-suspension scheduling policy by asking how to pause waste collection.

A similar topic, however, does not guarantee an answer. A question about transferring task amounts during contract renewal needs the conditions and restrictions, rather than a general guide to copying contracts. Semantic similarity alone could rank other renewal documents first and leave the required passage outside the top-k candidates. That distinction mattered when preparing evidence for QA.

I added sparse retrieval to compare terms actually present in the question and documents. BGE-M3 generates both representations; Qdrant searches them separately and combines their rankings with RRF. Rather than adding scores with different scales and distributions, RRF uses the position of each result. Dense retrieval helps with different wording, while sparse retrieval adds a route for specific terms such as "task amount" and "transfer."

Matching terms still does not establish whether a passage explains the required conditions. A cross-encoder compares the combined candidates with the question and reassesses relevance. Reranking cannot recover a document outside the candidate set, so the two searches first reduce omissions and reranking then refines the order. This expensive comparison is limited to retrieved candidates.

I also restricted the retrieval context. Repository, path, and branch filters prevent code from another repository or an in-development branch from being mistaken for current behavior. Lookups with a known target, such as a ticket number or function name, use a separate keyword-search path without embedding or reranking.

Similar chunks from one document could crowd out related policy and code. I limited results per document and reserved source-specific candidates so questions requiring comparison retained more than one type of evidence.

The 3D distribution below illustrates how dense and sparse vectors represent the same material differently. Its hybrid view concatenates normalized dense vectors with normalized, dimension-reduced sparse vectors for visualization only. Production retrieval combines rankings with RRF; distances and positions in the plot are not measurements of retrieval accuracy.

Limits of Traditional RAG

Limits Revealed After Connecting Retrieval and Generation

I connected Qwen3.5 to this retrieval foundation to generate answers grounded in the retrieved material. Testing the RAG system revealed several limitations.

First, answer quality depended heavily on the initial search results. Even when relevant documents were retrieved, I still had to determine whether they covered the conditions and exceptions needed to answer the question. When evidence was insufficient or sources gave conflicting explanations, the query needed to be revised and other material retrieved to fill the gaps.

Questions requiring information from several sources exposed further limitations. They needed to be broken into smaller questions, with evidence retrieved for each and subsequent searches adapted to what had already been found. A pipeline that connected retrieval and generation in a predefined sequence required this evaluation and further retrieval to be designed separately.

Increasing retrieval volume and passing more material at once also had limits. Unnecessary or duplicate content made useful evidence harder to identify and increased processing demands. Running a larger model ourselves to improve the interpretation of search results and answer quality was also constrained by available infrastructure and operating costs.

Agentic RAG Design

Letting Agents Retrieve the Context Each Task Needs

To address these limitations, I moved to Agentic RAG, incorporating question decomposition, evaluation of retrieved evidence, and further searches to fill information gaps.

I exposed the existing Retriever's search capabilities as MCP tools and assigned the Generator role, interpreting evidence and constructing answers, to a Claude agent. Document search and code lookup became tools available to the same agent, so a clue in one source could guide a lookup in another. I reused the Claude Code runtime for tool selection and conversation-context management, while the internal infrastructure handles knowledge collection, indexing, and retrieval.

Applying the Knowledge

From QA Preparation to Execution

I first applied the retrieved material to QA. I connected RAG to Codex and Claude through MCP so they could retrieve requirements, past bugs, code, and verification records together. Using this evidence, I built test cases covering behavior across roles and states, boundary conditions, exceptions, and regressions in related features.

I worked with the agents to quickly generate test cases and the data needed to run them, execute workflows in a real browser, and verify saved results and states through APIs. Testing different conditions and data combinations uncovered additional bugs I had missed during manual QA and broadened the range of scenarios I could test.

E2E handled repetitive checks of basic functionality. For new and changed features, I made the final judgments on policies and exceptions using the evidence retrieved by the agents and the execution results. Earlier reproduction conditions and the intent of a fix also informed retesting.

For automated basic checks and per-PR verification environments, see E2E QA Automation.

Expansion #1

From Internal Evaluation to Developer Adoption

To make the knowledge used in QA useful during development as well, I added capabilities for checking function usage, dependencies, and the impact of code changes. Developers could read the current code and form a change plan, then cross-check related requirements, past bugs, and consumers in other repositories.

Before introducing it into development workflows, I ran an internal A/B evaluation using reconstructed backend tickets that had shipped. The baseline was compared with an environment combining RAG, shared agent instructions, and five static code-structure tools. Implementation correctness rose from 79.5 to 84.8 for Opus and from 78.2 to 82.6 for Sonnet, improvements of 6.67% and 5.63%, respectively.

Both models implemented the same tickets in isolated environments using pre-change code and knowledge available at the time. Three independent Opus evaluation agents scored the outputs against five criteria: requirements met, related features and exception handling covered, past follow-up bugs prevented, unnecessary changes introduced, and regression risks from overengineering. Output tokens increased to about 1.22 times the baseline for Opus and 1.16 times for Sonnet, so I examined changes in generation volume alongside implementation correctness.

After confirming the improvement, I distributed the MCP connection settings and shared agent instructions to developers. The instructions explain when to retrieve internal policies and change history, and how to apply the evidence to code changes and verification. Developers could then use the same knowledge in the Claude, Cursor, and Codex tools they already used.

Expansion #2

Checking Missing Conditions in Plans Before Implementation

Even when developers implemented the requirements correctly, missing conditions or contradictions in the plan could still lead to bugs. I therefore extended the platform to review policies in Jira planning tickets before implementation. A planner requests a review with #qa-check in a Jira comment, and the worker posts the result to the same ticket.

The worker collects comments, field history, and directly related tickets so decisions made during discussion are not missed by reading only the description. Linked PRs are compared using synchronized PR head code. Unavailable code and an unestablished base branch remain explicit unknowns.

Code serves as evidence, but findings are written for decisions the planner can make. Rather than specifying which function to edit, the review prioritizes missing policy conditions and decisions needed, with priorities and source links. Planners use these findings to strengthen the policy.

The review model has read-only tools; only the worker can post Jira comments. Results retain the request comment's visibility. SQLite persists request IDs and progress so restarts and retries do not post the same result repeatedly.

Expansion #3

Asking About Internal Knowledge from the Work Page

I built a web UI and a Chrome extension so people could ask about internal knowledge without configuring MCP themselves. The web UI supports questions and conversation history, while the extension opens a question panel over the work page being viewed, displaying answers and their sources.

Both clients connect to the same answer layer, sharing the foundation through which a Claude agent retrieves the documents and code it needs. This extended access to internal knowledge from development tools to everyday work pages.

Operational Monitoring and Response

Monitoring RAG Knowledge Updates and Service Health

Using the same knowledge in QA, development, and planning raised another question: is the retrieved code deployed to the test environment? Code and commit lookups cannot answer that, so I connected separate read-only tools to check deployment versions and database change history.

I also made source freshness explicit. Retrieval and periodic indexing run separately, with stable document IDs for incremental synchronization. Source update status, baseline branches, and unmerged development branches remain distinct so old evidence and work in progress are not mistaken for the current state.

Investigating slow responses required relating retrieval behavior to resource use. I collected metrics from the Go retrieval server and Jira worker with Prometheus and built Grafana dashboards. Requests by tool and p50 and p95 latency can be compared with RSS, Go heap, goroutines, open file descriptors, and CPU, memory, and GPU usage. Indexing checkpoints and branch updates are shown separately.

A shared lock coordinates embedding and reranking on one GPU. If reranking fails, the service falls back to another model or the fused retrieval order, preventing a failure in detailed relevance scoring from failing the entire search.

A running Jira worker alone does not establish whether reviews are progressing. I separated the last successful poll, active reviews, job heartbeats, and Claude response activity to distinguish request-collection failures from stalled reviews. Polling runs independently of review execution, and failed jobs are limited to five total attempts, including the first.

I documented a runbook to turn these observations into response steps. Retrieval latency, error rate, index freshness, and Jira worker health serve as key service-level indicators (SLIs). Observed behavior, timeouts, polling intervals, and retry settings inform the checks and the order of response.

Outcomes

A Shared Knowledge Foundation for QA, Development, and Planning

I built an internal Agentic RAG platform from scratch (0→1) using Go and Python. I connected collection, indexing, and retrieval of Jira, Confluence, and Git material in one workflow, with agents using MCP for additional document and code lookups as needed.

In QA, policies and code retrieved by agents helped me quickly prepare test cases and data and uncover bugs I had missed during manual verification. After an internal A/B evaluation showed implementation correctness improvements of 6.67% for Opus and 5.63% for Sonnet, I introduced development support by sharing MCP settings and shared agent instructions with developers.

I also implemented the Jira policy review worker, web UI, and Chrome extension. Planners can have missing conditions and contradictions reviewed within their tickets, while the web UI and work-page panel let people ask about internal knowledge and check the sources behind the answers. These interfaces make the same retrieval foundation available in development tools, planning tickets, and the browser.

I built a Prometheus and Grafana monitoring environment for retrieval performance, resource use, knowledge updates, and Jira review status. I implemented fallback handling in the retrieval service, retries and persistent state in the review worker, and documented checks and response procedures in a runbook.

With E2E, test cases, RAG-assisted verification, and Jira follow-up working together, average verification volume rose from 29.2 tickets per day after E2E adoption to 33.1 after RAG adoption, while the post-release bug ratio fell from 3.9% to 2.9%. Across regular releases after RAG adoption (2.16.2–2.22.0), post-release bugs decreased to 23 among 794 tickets in scope for QA.