A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs

Kevin Peyton, Saritha Unnikrishnan

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)

Abstract

Online learning enables academic institutions to accommodate increased student numbers at scale. With this scale comes high demands on support staff for help in dealing with general questions relating to qualifications and registration. Chatbots that implement Frequently Asked Questions (FAQs) can be a valuable part in this support process. A chatbot can provide constant availability in answering common questions, allowing support staff to engage on higher value one-to-one communication with prospective students. A variety of approaches can be used to create these chatbots including vertical platforms, frameworks, and direct model implementation. A comparative analysis is required to establish which approach provides the most accuracy for an existing, available dataset. This paper compares intent classification results of two popular chatbot frameworks to a state-of-the-art Sentence BERT (SBERT) model that can be used to build a robust chatbot. A methodology is outlined which includes the preparation of a university FAQ dataset into a chatbot friendly format for upload and training of each implementation. Results obtained from the framework-based implementations are generated using their published Application Programming Interfaces (APIs). This enables intent classification using testing phrases and finally comparison of F1 scores. Using ten intents comprising 284 training phrases and 85 testing phrases it was found that a SBERT model outperformed all others with an F1-score of 0.99. Initial comparison with the literature suggests that the F1-scores obtained for Google Dialogflow (0.96) and Microsoft QnA Maker (0.95) are very similar to other benchmarking exercises where NLU (Natural Language Understanding) has been compared.

Original languageEnglish
Article number100856
JournalResults in Engineering
Volume17
DOIs
Publication statusPublished - Mar 2023

Keywords

  • Chatbots
  • FAQs
  • Natural language understanding
  • Online learning
  • SBERT

Fingerprint

Dive into the research topics of 'A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs'. Together they form a unique fingerprint.

Cite this