Case study AI demo
Lease pre-check assistant
An adviser pastes a customer's WhatsApp message. Seconds later: the car, whether it fits the policy, the monthly payment and a draft reply.
- Role
- Creator. Designed and built as a demo for a lease company.
- Owned
- Use case, Agent and tools, Rules engine, RAG, Evals, Interface
- Stack
- Python, FastAPI, LangChain, Anthropic API, Voyage embeddings, RDW open data, React, TypeScript, Vercel
- Status
- Demo. The policy and handbook are made up for it.
01 The problem
Four small steps before an adviser can reply.
A business owner sends a licence plate, a price and a term: “Can I lease the Audi H-702-BH for 42,000 over 48 months?” Before the adviser can answer, they look the car up, check the acceptance policy, calculate the monthly payment and write a message. None of it is hard, but together it takes time. And the part that matters most to the customer is “not like this, but like this it works”.
02 The key decision
The model doesn't calculate and doesn't decide.
A language model is good at language: understanding a messy message, pulling out the details, explaining an outcome clearly. It is not reliable at arithmetic, and it can forget a rule that only lives in a prompt. For a financial decision that isn’t acceptable.
So the acceptance rules and the annuity calculation are plain Python with tests, and the model calls them as tools. The verdict and the monthly payment in the case file come straight from the tool results, not from the model’s text. Changing the policy means changing a constant and running the tests, not rewriting a prompt and hoping.
- Input
- Customer messageDutch or English
- Agent
- Claude via LangChainpicks the tool it needs
- Tools
- zoek_voertuigfinds the car in RDW open data
- controleer_leaserules and monthly payment, in tested code
- zoek_in_beleidhybrid RAG over the handbook
- Output
- Answer
- Draft reply
- Case file
When the adviser changes the term or the down payment, the check runs again without the model: faster, free and the same answer every time.
03 RAG
Measured, not guessed.
Not every message is about a car. For questions about conditions, the assistant searches a handbook of six documents and may only answer from what it finds, with the source shown. If it isn’t there, it refers to the acceptance team.
split by heading01 Cut where the policy already cuts
Every section of the handbook becomes one chunk, with its heading. Cutting at a fixed length can split a rule in half. Six documents, 51 chunks.
BM25 first02 Start simple, then measure
32 questions written the way customers ask them. Keyword search alone found the right section in the top 3 for 56%. Almost every miss was a synonym: 'zzp' never found the section on starters.
BM25 + embeddings03 Then go hybrid
Embeddings search by meaning, reciprocal rank fusion merges both rankings. Without an embeddings key it falls back to BM25, so it always works.
When a question goes wrong, I fix the knowledge base rather than the prompt, and the question goes into the eval.
04 Built like production
Evals, tests and limits.
A retrieval eval checks whether the right section is found, and an extraction eval checks whether the agent pulls the right details from a message. The rules, the calculation and the API are covered by automated tests, with the RDW, the embeddings and the model mocked. The demo sits behind a password checked on the server, with a limit per IP, capped message length and history, a cap on output tokens and a spending limit at the provider.
05 Next steps
What I would do with real data.
Real, anonymised customer messages in the evals, with the evals running in CI. A reranker on the top ten once the handbook grows. Open recalls and the mileage verdict from the RDW in the check. And messages coming in through the WhatsApp Business Platform, with the adviser always as the last step: the assistant deliberately never sends anything to a customer itself.