Skip to main content
Blog

SwissText 2023

Adapting Large Language Models for Customer Request Handling: An exploration of possible approaches

Katsiaryna Mlynchyk, Alexandros Paramythis

Challenge

“Adapt” LLMs to the domain at hand, and to the factual knowledge of the individual company / organization for automated handling of customer requests – received through channels such as email, chat, social media, etc. 

Such models will need to have access, in one way or another, to privileged, non-public information in the company’s knowledge base.

Requirements

  • can be trained / fine-tuned / otherwise adapted with reasonable resources
  • can be prepared and used on-premise, to avoid sending private information outside  the company’s infrastructure
  • must support the language in use

Exploration of possible approaches

Watch the video of our presentation at SwissText 2023, in which we present in brief different approaches of evaluating LLMs and their potential to “adapt” to the domain at hand. Alternatively, you can browse through the presentation slides that follow below.

Big picture

Big picture of Adapting Large Language Models for Customer Request Handling

Banking Use Case: Raiffeisenlandesbank OÖ

LLM Adaptation for Banking Use Case - Raiffeisenlandesbank OÖ

General data

  • Dolly v2 
    • 15k instructions translated to DE
  • Open Assistant
    • 4k “chat trees” either originally in DE, or translated to DE
  • Translated datasets (will be made available on Hugging Face after SwissText)

Project-specific data

  • FAQs
  • Web site content
  • Test emails

Considered models

Considered LLMs for customer request handling in banking use case

Simple QA

Testing Large Language Models with simple banking question answering

Process chain for QA (over FAQ / over all web data)

Process chain for question answering (over FAQ / over all web data)

Possible sources of errors and their investigation

Possible sources of errors during LLM adaptation experiments and their investigation

Test email generation

Process of creating test email data for the evaluation of Large Language Models

Evaluation & Results

Comparison from tests with “given context”

Evaluation and comparison of LLMs with given context - part 1
Evaluation and comparison of LLMs with given context - part 2

Base models’ comparison

Evaluation & Results of Base Large Language Models

More results coming soon…

Download data

Translated data

Automatically generated test emails (appr. 350) with Openai

Raw QA on FAQ

QA on all data

Generated E-mails (with given context)

Interested in finding out more about our evaluation?

Contact us