Assessment of Retrieval and Generative Chatbots in Tourism Information Service

Sharfina Febbi Handayani; Dairoh Dairoh; Dwi Intan Af'idah

doi:10.30595/juita.v13i1.24182

Authors

Sharfina Febbi Handayani Politeknik Harapan Bersama
Dairoh Dairoh Politeknik Harapan Bersama
Dwi Intan Af'idah Politeknik Harapan Bersama

DOI:

https://doi.org/10.30595/juita.v13i1.24182

Keywords:

Chatbot, GEMMA, TLLaMS2, Tourism.

Abstract

Chatbots are essential for improving the customer experience on tourism websites, especially when it comes to arranging travel and offering precise information. The purpose of this study is to evaluate the effectiveness of generative and retrieval-based chatbots in the tourism information service. Two retrieval-based models are MLP-based single QA and multi QA and two generative-based models namely LLaMA 2 and GEMMA were evaluated using confusion matrix, BLUE score, correctness response and naturalness response. The study found that LLaMA 2 outperformed other models, with the highest response Accuracy of 0.89, naturalness of 0.75, and BLEU score of 0.33. GEMMA received the lowest score, suggesting that it has trouble coming up with precise and organic answers. The retrieval-based models showed strong accuracy but were less natural in their responses. The ease of dataset creation for generative models, which only requires narrative text, further positions LLaMA 2 as the most suitable option for improving user experience in Tegal tourism services.

Author Biographies

Sharfina Febbi Handayani, Politeknik Harapan Bersama

Informatics Engineering

Dairoh Dairoh, Politeknik Harapan Bersama

Informatics Engineering

Dwi Intan Af'idah, Politeknik Harapan Bersama

Informatics Engineering

References

[1] T. Pencarelli, “The digital revolution in the travel and tourism industry,” Inf. Technol. Tour., vol. 22, no. 3, pp. 455–476, 2020, doi: 10.1007/s40558-019-00160-3.

[2] C. C. Lin, W. Y. Liu, and Y. W. Lu, “Three-dimensional internet-of-things deployment with optimal management service benefits for smart tourism services in forest recreation parks,” IEEE Access, vol. 7, pp. 182366–182380, 2019, doi: 10.1109/ACCESS.2019.2960212.

[3] J. R. Chang, M. Y. Chen, L. S. Chen, and S. C. Tseng, “Why Customers Don’t Revisit in Tourism and Hospitality Industry?,” IEEE Access, vol. 7, pp. 146588–146606, 2019, doi: 10.1109/ACCESS.2019.2946168.

[4] D. I. Af’Idah, Dairoh, and S. F. Handayani, “Virtual Tour Application as A Tourism Information Media Using Multimedia Development Life Cycle,” in 2024 1st International Conference on Robotics, Engineering, Science, and Technology (RESTCON), 2024, pp. 207–213. doi: 10.1109/RESTCON60981.2024.10463581.

[5] E. Adamopoulou and L. Moussiades, “Chatbots: History, technology, and applications,” Mach. Learn. with Appl., vol. 2, no. July, p. 100006, 2020, doi: 10.1016/j.mlwa.2020.100006.

[6] R. Dsouza, S. Sahu, R. Patil, and D. R. Kalbande, “Chat with Bots Intelligently: A Critical Review Analysis,” 2019 6th IEEE Int. Conf. Adv. Comput. Commun. Control. ICAC3 2019, 2019, doi: 10.1109/ICAC347590.2019.9036844.

[7] D. R. Darmawan and R. Arifudin, “Enhancing Durrotalk Chatbot Accuracy Utilizing a Hybrid Model Based on Recurrent Neural Network (RNN) Algorithm and Decision Tree,” JUITA J. Inform., vol. 12, no. 1, p. 81, 2024, doi: 10.30595/juita.v12i1.20868.

[8] L. Benaddi, C. Ouaddi, A. Jakimi, and B. Ouchao, “Towards A Software Factory for Developing the Chatbots in Smart Tourism Mobile Applications,” Procedia Comput. Sci., vol. 231, no. 2023, pp. 275–280, 2024, doi: 10.1016/j.procs.2023.12.203.

[9] O. M. Alyasiri, K. Selvaraj, H. A. Younis, T. Mueen Sahib, M. F. Almasoodi, and I. M. Hayder, “A Survey on the Potential of Artificial Intelligence Tools in Tourism Information Services Keywords Artificial Intelligence ChatGPT Hospitality and Tourism Industry Tourism Education and Research Tourism and Management,” Babylonian J. Artif. Intell., vol. 2024, pp. 1–8, 2024.

[10] P. J. Antony, R. Kannan, and A. Professor, “Revolutionizing the Tourism Industry through Artificial Intelligence: A Comprehensive Review of AI Integration, Impact on Customer Experience, Operational Efficiency, and Future Trends,” www.chandigarhphilosophers.com International Journal for Multidimensional Research Perspectives (IJMRP, vol. ISSN, no. 1. pp. 2584–2613, 2024.

[11] D. I. Af’idah, P. D. Anggraeni, M. Rizki, A. B. Setiawan, and S. F. Handayani, “Aspect-Based Sentiment Analysis for Indonesian Tourist Attraction Reviews Using Bidirectional Long Short-Term Memory,” JUITA J. Inform., vol. 11, no. 1, p. 27, 2023, doi: 10.30595/juita.v11i1.15341.

[12] H. Akkineni, P. V. S. Lakshmi, and L. Sarada, “Design and Development of Retrieval-Based Chatbot Using Sentence Similarity BT - IoT and Analytics for Sensor Networks,” P. Nayak, S. Pal, and S.-L. Peng, Eds., Singapore: Springer Singapore, 2022, pp. 477–487.

[13] B. El Bakkouri, S. Raki, and T. Belgnaoui, “The Role of Chatbots in Enhancing Customer Experience: Literature Review,” Procedia Comput. Sci., vol. 203, pp. 432–437, 2022, doi: 10.1016/j.procs.2022.07.057.

[14] T. B. Brown et al., “Language models are few-shot learners,” Adv. Neural Inf. Process. Syst., vol. 2020-Decem, 2020.

[15] N. Mathur, T. Baldwin, and T. Cohn, “Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics,” Proc. Annu. Meet. Assoc. Comput. Linguist., pp. 4984–4997, 2020, doi: 10.18653/v1/2020.acl-main.448.

[16] M. K. Ogirala, R. Tallapaneni, S. M. Chalamcharla, and A. Chinta, “A Medical Diagnosis and Treatment Recommendation Chatbot using MLP,” in 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), 2023, pp. 495–500. doi: 10.1109/ICAAIC56838.2023.10141211.

[17] S. Pandey and S. Sharma, “A comparative study of retrieval-based and generative-based chatbots using Deep Learning and Machine Learning,” Healthc. Anal., vol. 3, no. May, p. 100198, 2023, doi: 10.1016/j.health.2023.100198.

[18] G. Sperlí, “A cultural heritage framework using a Deep Learning based Chatbot for supporting tourist journey,” Expert Syst. Appl., vol. 183, no. May, p. 115277, 2021, doi: 10.1016/j.eswa.2021.115277.

[19] H. Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,” 2023.

[20] “GPT-4 Technical Report,” vol. 4, pp. 1–100, 2023.

[21] Gemma Team et al., “Gemma: Open Models Based on Gemini Research and Technology,” 2024.

[22] K. Adnan and R. Akbar, “An analytical study of information extraction from unstructured and multidimensional big data,” J. Big Data, vol. 6, Oct. 2019, doi: 10.1186/s40537-019-0254-8.

[23] D. Cabezas, R. Fonseca-Delgado, I. Reyes-Chacón, P. Vizcaino-Imacaña, and M. Morocho-Cayamcela, “Integrating a LLaMa-based Chatbot with Augmented Retrieval Generation as a Complementary Educational Tool for High School and College Students,” no. July, pp. 395–402, 2024, doi: 10.5220/0012763000003753.

[24] C. Wu, W. Lin, X. Zhang, Y. Zhang, W. Xie, and Y. Wang, “PMC-LLaMA: toward building open-source language models for medicine,” J. Am. Med. Informatics Assoc., 2024, doi: 10.1093/jamia/ocae045.

[25] H. Yin, A. Aryani, and N. Nambiar, “Evaluating the Performance of Large Language Models for SDG Mapping (Technical Report),” pp. 1–6, 2024.

[26] M. Hinck, M. L. Olson, D. Cobbley, S.-Y. Tseng, and V. Lal, “LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model,” 2024.

[27] O. Ridge and O. Ridge, “Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation,” pp. 1–13.

[28] A. Kumar, “Towards Optimal NLP Solutions : Analyzing GPT and LLaMA-2 Models Across Model Scale , Dataset Size , and Task Diversity,” vol. 14, no. 3, pp. 14219–14224, 2024.