System Usability and Design Evaluation of AI Chatbots: A Comparative Analysis of ChatGPT, Google Bard, and Bing Chat

Abstract

Artificial intelligence (AI) has brought significant advancements in technology while the chatbots like ChatGPT, Google Bard, and Bing Chat are some of its remarkable innovations. These chatbots are helping users with diverse backgrounds by generating ideas, providing resources, and overall knowledge management. We acknowledge that these chatbots are still in their experimental stages of use. Evaluating the usability and user experience of chatbots becomes crucial to make them more usable, accessible, and intuitive to end users around the globe. Thus, the objectives of this research are to make a comparative usability analysis of AI-generated chatbots: Google Bard, ChatGPT, and Bing Chat. To achieve these goals, firstly, the System Usability Score (SUS) through questionnaire surveys and secondly, Heuristic Evaluation (HE) through expert observation were used. Through HE, we investigated characteristics of design, user engagement, and some other specific usability lacking along with a severity score that suggests both urgent and gradual usability improvement action. As an outcome, this study found that the SUS evaluation provided a comprehensive view of user satisfaction. Google Bard and Bing Chat received lower SUS scores, while ChatGPT demonstrated comparatively better usability, with a SUS score above 70. Again, a comparative usability analysis of AI-generated chatbots (ChatGPT, Google Bard and Bing Chat) reveals that, while all these applications suffer from a notable number of usability problems, ChatGPT demonstrates better usability performance compared to Google Bard and Bing Chat.

MIJST, Vol. 13, June 2025 : 83-97

DOI: https://doi.org/10.47981/j.mijst.13(01)2025.522(83-97)