![](https://www.contexity.ch/wp-content/uploads/swisstext-2023.jpeg)
Adapting Large Language Models for Customer Request Handling: An exploration of possible approaches
Katsiaryna Mlynchyk, Alexandros Paramythis
Challenge
“Adapt” LLMs to the domain at hand, and to the factual knowledge of the individual company / organization for automated handling of customer requests – received through channels such as email, chat, social media, etc.
Such models will need to have access, in one way or another, to privileged, non-public information in the company’s knowledge base.
Requirements
- can be trained / fine-tuned / otherwise adapted with reasonable resources
- can be prepared and used on-premise, to avoid sending private information outside the company’s infrastructure
- must support the language in use
Exploration of possible approaches
Watch the video of our presentation at SwissText 2023, in which we present in brief different approaches of evaluating LLMs and their potential to “adapt” to the domain at hand. Alternatively, you can browse through the presentation slides that follow below.
Big picture
![Big picture of Adapting Large Language Models for Customer Request Handling](https://www.contexity.ch/wp-content/uploads/swisstext2023-big-picture-768x416.png)
Banking Use Case: Raiffeisenlandesbank OÖ
![LLM Adaptation for Banking Use Case - Raiffeisenlandesbank OÖ](https://www.contexity.ch/wp-content/uploads/swisstext2023-use-case.jpg)
General data
- Dolly v2
- 15k instructions translated to DE
- Open Assistant
- 4k “chat trees” either originally in DE, or translated to DE
- Translated datasets (will be made available on Hugging Face after SwissText)
- Get in touch to receive a notification: paramythis@contexity.ch
Project-specific data
- FAQs
- Web site content
- Test emails
Considered models
![Considered LLMs for customer request handling in banking use case](https://www.contexity.ch/wp-content/uploads/swisstext2023-considered-models.png)
Simple QA
![Testing Large Language Models with simple banking question answering](https://www.contexity.ch/wp-content/uploads/swisstext2023-simple-qa.png)
Process chain for QA (over FAQ / over all web data)
![Process chain for question answering (over FAQ / over all web data)](https://www.contexity.ch/wp-content/uploads/swisstext2023-process-chain-qa.png)
Possible sources of errors and their investigation
![Possible sources of errors during LLM adaptation experiments and their investigation](https://www.contexity.ch/wp-content/uploads/swisstext2023-sources-of-error-and-investigation.jpg)
Test email generation
![Process of creating test email data for the evaluation of Large Language Models](https://www.contexity.ch/wp-content/uploads/swisstext2023-creating-test-data.png)
Evaluation & Results
Comparison from tests with “given context”
![Evaluation and comparison of LLMs with given context - part 1](https://www.contexity.ch/wp-content/uploads/swisstext-2023-comparison-models.png)
![Evaluation and comparison of LLMs with given context - part 2](https://www.contexity.ch/wp-content/uploads/swisstext-2023-comparison-igel.png)
Base models’ comparison
![Evaluation & Results of Base Large Language Models](https://www.contexity.ch/wp-content/uploads/swisstext-2023-comparison-base-models.png)
More results coming soon…
Download data
Translated data
Automatically generated test emails (appr. 350) with Openai
Raw QA on FAQ
QA on all data
Generated E-mails (with given context)