A RAG chatbot people could actually trust
At Blenheim Chalcot I designed a retrieval-augmented generation (RAG) chatbot to automate information retrieval, measured on accuracy first and speed second.
RoleAssociate PM
CompanyBlenheim Chalcot India
When2024 – 2025
Outcome95% accuracy
The problem
People spent too long finding answers that already existed somewhere in documentation. A plain chatbot would answer quickly but could make things up — and one confident wrong answer is enough for people to stop trusting the tool.
Approach
- Ground every answer in sources. RAG retrieves the relevant documents first and has the model answer only from them, which keeps answers traceable.
- Define "correct" before building. Response accuracy was the main success metric, checked against real questions, with response time as the second.
- Start from the workflow. This was one of 10+ high-friction manual workflows I identified and assessed for AI automation; the validated ones became PRDs for engineering.
Results
- 95% response accuracy
- 20% faster query response time
Related work
In the same role I led an AI-agent proof of concept for digital marketing that mapped manual content planning into automated workflows, cutting content creation turnaround by 65%.
What I learned
With LLM products, trust is the product. Agreeing on how correctness is measured up front made every later decision — retrieval, prompts, scope — much easier to make.