Skip to main content
News

SwissText 2023

Adapting Large Language Models for Customer Request Handling: An exploration of possible approaches

Katsiaryna Mlynchyk, Alexandros Paramythis

Challenge

“Adapt” LLMs to the domain at hand, and to the factual knowledge of the individual company / organization for automated handling of customer requests – received through channels such as email, chat, social media, etc. 

Such models will need to have access, in one way or another, to privileged, non-public information in the company’s knowledge base.

Requirements

  • can be trained / fine-tuned / otherwise adapted with reasonable resources
  • can be prepared and used on-premise, to avoid sending private information outside  the company’s infrastructure
  • must support the language in use

Big picture

Banking Use Case: Raiffeisenlandesbank OÖ

General data

  • Dolly v2 
    • 15k instructions translated to DE
  • Open Assistant
    • 4k “chat trees” either originally in DE, or translated to DE
  • Translated datasets (will be made available on Hugging Face after SwissText)

Project-specific data

  • FAQs
  • Web site content
  • Test emails

Considered models

Simple QA

Process chain for QA (over FAQ / over all web data)

Possible sources of errors and their investigation

Test email generation

Evaluation & Results

Comparison from tests with “given context”

Base models’ comparison

More results coming soon…

Download data

Translated data

Automatically generated test emails (appr. 350) with Openai

Raw QA on FAQ

QA on all data

Generated E-mails (with given context)