TY - JOUR
T1 - A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs
AU - Peyton, Kevin
AU - Unnikrishnan, Saritha
N1 - Publisher Copyright:
© 2022 The Authors
PY - 2023/3
Y1 - 2023/3
N2 - Online learning enables academic institutions to accommodate increased student numbers at scale. With this scale comes high demands on support staff for help in dealing with general questions relating to qualifications and registration. Chatbots that implement Frequently Asked Questions (FAQs) can be a valuable part in this support process. A chatbot can provide constant availability in answering common questions, allowing support staff to engage on higher value one-to-one communication with prospective students. A variety of approaches can be used to create these chatbots including vertical platforms, frameworks, and direct model implementation. A comparative analysis is required to establish which approach provides the most accuracy for an existing, available dataset. This paper compares intent classification results of two popular chatbot frameworks to a state-of-the-art Sentence BERT (SBERT) model that can be used to build a robust chatbot. A methodology is outlined which includes the preparation of a university FAQ dataset into a chatbot friendly format for upload and training of each implementation. Results obtained from the framework-based implementations are generated using their published Application Programming Interfaces (APIs). This enables intent classification using testing phrases and finally comparison of F1 scores. Using ten intents comprising 284 training phrases and 85 testing phrases it was found that a SBERT model outperformed all others with an F1-score of 0.99. Initial comparison with the literature suggests that the F1-scores obtained for Google Dialogflow (0.96) and Microsoft QnA Maker (0.95) are very similar to other benchmarking exercises where NLU (Natural Language Understanding) has been compared.
AB - Online learning enables academic institutions to accommodate increased student numbers at scale. With this scale comes high demands on support staff for help in dealing with general questions relating to qualifications and registration. Chatbots that implement Frequently Asked Questions (FAQs) can be a valuable part in this support process. A chatbot can provide constant availability in answering common questions, allowing support staff to engage on higher value one-to-one communication with prospective students. A variety of approaches can be used to create these chatbots including vertical platforms, frameworks, and direct model implementation. A comparative analysis is required to establish which approach provides the most accuracy for an existing, available dataset. This paper compares intent classification results of two popular chatbot frameworks to a state-of-the-art Sentence BERT (SBERT) model that can be used to build a robust chatbot. A methodology is outlined which includes the preparation of a university FAQ dataset into a chatbot friendly format for upload and training of each implementation. Results obtained from the framework-based implementations are generated using their published Application Programming Interfaces (APIs). This enables intent classification using testing phrases and finally comparison of F1 scores. Using ten intents comprising 284 training phrases and 85 testing phrases it was found that a SBERT model outperformed all others with an F1-score of 0.99. Initial comparison with the literature suggests that the F1-scores obtained for Google Dialogflow (0.96) and Microsoft QnA Maker (0.95) are very similar to other benchmarking exercises where NLU (Natural Language Understanding) has been compared.
KW - Chatbots
KW - FAQs
KW - Natural language understanding
KW - Online learning
KW - SBERT
UR - http://www.scopus.com/inward/record.url?scp=85145973734&partnerID=8YFLogxK
U2 - 10.1016/j.rineng.2022.100856
DO - 10.1016/j.rineng.2022.100856
M3 - Article
AN - SCOPUS:85145973734
SN - 2590-1230
VL - 17
JO - Results in Engineering
JF - Results in Engineering
M1 - 100856
ER -