Artificial Intelligence is rapidly transforming financial services, helping organizations automate support, improve customer experiences, and make better business decisions. Yet, many believe that achieving better AI performance simply requires upgrading to a larger or more advanced language model.
Our experience tells a different story.
Over the past quarter, we rebuilt the AI assistant used daily by our support, sales, and marketing teams. Instead of replacing the AI model, we focused on improving the quality of the knowledge it relies on.
The results were significant.
Technical support issues involving complex trading and settlement processes that previously took weeks to resolve can now be addressed in minutes. Some internal processing workflows now run up to 18 times faster, and our knowledge base continues to improve with every interaction.
This article highlights the five key lessons we learned throughout the process.
Lesson 1: Your Dataset Is the Product
One of the biggest misconceptions about AI is that the model alone determines performance.
In reality, the quality of the data behind the model often has a much greater impact.
We conducted a complete audit of more than 1,100 knowledge articles created by our subject matter experts.
What we discovered surprised us.
Nearly 20% of the knowledge entries were duplicates or near-duplicates, and one section of the knowledge base was 75% redundant, with multiple experts answering the same question in slightly different ways.
Although technically correct, these duplicate articles confused the retrieval system and reduced the AI’s ability to consistently find the best answer.
Rather than generating new content, we:
- Removed duplicate articles
- Consolidated overlapping information
- Rewrote unclear entries
- Revalidated every article for quality and accuracy
The result was a dramatic improvement in retrieval accuracy—from 82% to 100%—without making any changes to the underlying AI model.
The lesson is simple:
Good AI begins with great knowledge.
Lesson 2: Questions Matter More Than Answers
Traditional documentation is written for people reading from start to finish.
Support engineers don’t work that way.
When customers encounter issues, they search using:
- Error messages
- Trade rejection codes
- Symptoms
- Transaction references
- Partial system messages
Instead of writing articles like technical manuals, we redesigned our knowledge base around how real users actually search.
The answers themselves didn’t change.
The questions did.
By restructuring articles to match real-world search behavior, retrieval performance improved significantly, allowing users to locate accurate answers much faster.
Sometimes improving AI is simply about understanding how people ask questions.
Lesson 3: More Context Doesn’t Always Mean Better Results
A common assumption in Retrieval-Augmented Generation (RAG) systems is that providing more context automatically produces better answers.
Our testing proved otherwise.
Initially, our AI retrieved 40 documents for every query.
After benchmarking different configurations, we discovered that retrieving just 10 carefully selected documents produced virtually identical recall while dramatically reducing unnecessary information.
The benefits included:
- Faster response times
- Lower processing costs
- Less irrelevant information
- More focused AI responses
In AI, more data is not always better.
The right data is.
Lesson 4: The Most Dangerous Bugs Don’t Always Produce Errors
One of the most important discoveries during our rebuild wasn’t an obvious system failure.
It was a hidden ranking issue.
Deep within the retrieval pipeline, a similarity score had effectively been interpreted as a distance score, causing the AI to prioritize less relevant documents over better ones.
Because another layer in the pipeline partially compensated for the issue, the system continued functioning.
No errors appeared.
No alarms were triggered.
Yet the AI consistently delivered mediocre responses.
This experience reinforced a valuable engineering principle:
If you aren’t measuring AI performance end-to-end against a benchmark, you’re relying on assumptions rather than evidence.
Continuous testing and evaluation are essential for maintaining reliable AI systems.
Lesson 5: Some of the Best Documentation Already Exists
Not all valuable knowledge comes from manuals or support tickets.
Some of the most useful documentation already existed inside our own software.
We extracted and documented 33 trade rejection codes directly from the validation engine, providing detailed explanations for each one, including:
- What the error means
- Why it occurs
- How to verify it
- How to resolve it
Support teams can now provide customers with accurate explanations in seconds instead of manually investigating every issue.
This approach transforms internal system intelligence into customer-facing knowledge.
Building an AI That Learns Continuously
Improving AI is not a one-time project.
It is an ongoing process.
Every piece of user feedback becomes an opportunity to improve the knowledge base.
When users indicate that an answer was unhelpful, that feedback becomes a candidate for a new or improved knowledge article.
Artificial Intelligence helps classify and evaluate the content, while human experts validate every addition before publication.
This quality-controlled feedback loop ensures that our AI becomes smarter over time while maintaining high standards of accuracy.
The Business Impact
Our investment in better knowledge engineering has already delivered measurable results:
- Technical support resolution times reduced from weeks to minutes.
- Retrieval accuracy improved from 82% to 100%.
- Some internal workflows now execute 18 times faster than before.
- Institutional knowledge is captured in a centralized system rather than remaining with individual employees.
- New team members can become productive faster through structured, searchable documentation.
These improvements benefit not only our internal teams but also the customers who rely on our solutions every day.
Preparing for the Future of Capital Markets
Nigeria’s capital market continues to evolve, bringing increased trading activity, new financial products, and growing customer expectations.
As transaction volumes increase, technology must scale alongside them.
At InfoWARE, we believe the future of AI in financial services isn’t defined by who has the largest model—it’s defined by who manages knowledge most effectively.
By combining high-quality data, continuous evaluation, intelligent retrieval, and expert validation, we’re building AI systems that deliver faster support, more accurate answers, and greater confidence for our customers.
The models will continue to improve.
Our competitive advantage will remain the quality of the knowledge behind them.
This article is based on our latest LinkedIn thought leadership post, where we share how better knowledge engineering—not a bigger AI model—helped transform support operations and improve customer outcomes.
👉 Read the original LinkedIn post and join the conversation:
[#ai #capitalmarkets #fintech #techleadership | Uwa Agbonile]
Follow InfoWARE as we continue sharing insights on Artificial Intelligence, Capital Markets, FinTech, and the technologies shaping the future of financial services.
