An artificial intelligence application in mathematics education: Evaluating ChatGPT’s academic achievement in a mathematics exam

Nezihe Korkmaz Guler; Zeynep Gul Dertli; Elif Boran; Bahadir Yildiz

doi:10.29333/pr/14145

Full Text (PDF)

Nezihe Korkmaz Guler ¹ ^* , Zeynep Gul Dertli ² , Elif Boran ¹ , Bahadir Yildiz ²

More Detail

¹ Çukurova University, Adana, TÜRKİYE² Hacettepe University, Ankara, TÜRKİYE^* Corresponding Author

Abstract

The aim of the research is to investigate the academic achievement of ChatGPT, an artificial intelligence based chatbot, in a national mathematics exam. For this purpose, 3.5 and 4 versions of ChatGPT were asked mathematics questions in a national exam. The method of the research is a case study. In the research, 3.5 and 4 versions of ChatGPT were used as data collection tools. The answers given by both versions of ChatGPT were analyzed separately by three researchers. As a result of the analysis of the data, it was found that ChatGPT-4 was more successful in the exam compared to ChatGPT-3,5 version, was better at understanding the questions asked, understood the instructions better and included more details in the question solution, and at the same time, both versions made common and different mistakes. According to the findings of the study, it was concluded that ChatGPT sometimes worked very well, sometimes only worked well and sometimes failed. In the light of the findings of the study, it can be suggested to use ChatGPT versions in mathematics education to obtain basic information and to get supervised help.

Keywords

artificial Intelligence
ChatGPT
mathematics education
achievement

License

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Article Type: Research Article

PEDAGOGICAL RES, Volume 9, Issue 2, April 2024, Article No: em0188

https://doi.org/10.29333/pr/14145

Publication date: 01 Apr 2024

Online publication date: 25 Jan 2024

Article Views: 1599

Article Downloads: 1113

Open Access References How to cite this article

References

Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 100006. https://doi.org/10.1016/j.mlwa.2020.100006
Bellegarda, J. R. (2004). Statistical language model adaptation: Review and perspectives. Speech Communication, 42(1), 93-108. https://doi.org/10.1016/j.specom.2003.08.002
Bozkurt, A., & Sharma, R. C. (2023). Generative AI and prompt engineering: The art of whispering to let the genie out of the algorithmic world. Asian Journal of Distance Education, 18(2). https://doi.org/10.5281/zenodo.8174941
Chassignol, M., Khoroshavin, A., Klimova, A., & Bilyatdinova, A. (2018). Artificial Intelligence trends in education: A narrative overview. Procedia Computer Science, 136, 16-24. https://doi.org/10.1016/j.procs.2018.08.233
Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510
Cooper, G. (2023). Examining science education in ChatGPT: An exploratory study of generative artificial intelligence. Journal of Science Education and Technology, 32(3), 444-452. https://doi.org/10.1007/s10956-023-10039-y
Coppin, B. (2004). Artificial intelligence illuminated. Jones & Bartlett Learning.
Creswell, J. W. (2021). Nitel araştırma yöntemleri [Qualitative research methods]. M. Butun, & S. B. Demir (Trans.), Siyasal Kitapevi [Political Bookstore].
Duan, Y., Edwards, J. S., & Dwivedi, Y. K. (2019). Artificial intelligence for decision making in the era of big data–Evolution, challenges and research agenda. International Journal of Information Management, 48, 63-71. https://doi.org/10.1016/j.ijinfomgt.2019.01.021
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A., Al-Busaidi, A., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., ... Wright, R. (2023). Opinion papr: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
French, R. M. (2000). The Turing test: The first 50 years. Trends in Cognitive Sciences, 4(3), 115-122. https://doi.org/10.1016/S1364-6613(00)01453-4
Gašević, D., Siemens, G., & Sadiq, S. (2023). Empowering learners for the age of artificial intelligence. Computers and Education: Artificial Intelligence, 4, 100130. https://doi.org/10.1016/j.caeai.2023.100130
Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). ChatGPT has aced the test of understanding in college economics: Now what? The American Economist, 68(2), 233-245. https://doi.org/10.1177/05694345231169654
Huh, S. (2023). Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: A descriptive study. Journal of Educational Evaluation for Health Professions, 20, 1. https://doi.org/10.3352/jeehp.2023.20.01
Hwang, G. J., & Tu, Y. F. (2021). Roles and research trends of artificial intelligence in mathematics education: A bibliometric mapping analysis and systematic review. Mathematics, 9(6), 584. https://doi.org/10.3390/math9060584
Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of Artificial Intelligence in Education. Computers and Education: Artificial Intelligence, 1, 100001. https://doi.org/10.1016/j.caeai.2020.100001
Hwang, S., Flavin, E., & Lee, J. E. (2023). Exploring research trends of technology use in mathematics education: A scoping review using topic modeling. Education and Information Technologies, 28, 10753-10780. https://doi.org/10.1007/s10639-023-11603-0
Kar, A. K. (2016). Bio inspired computing–A review of algorithms and scope of applications. Expert Systems with Applications, 59, 20-32. https://doi.org/10.1016/j.eswa.2016.04.018
Lancaster, T. (2023). Artificial intelligence, text generation tools and ChatGPT–Does digital watermarking offer a solution? International Journal for Educational Integrity, 19(1), 10. https://doi.org/10.1007/s40979-023-00131-6
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. https://doi.org/10.1038/nature14539
Lokman, A. S., & Ameedeen, M. A. (2019). Modern chatbot systems: A technical review. In Proceedings of the Future Technologies Conference 2018 (pp. 1012-1023). Springer. https://doi.org/10.1007/978-3-030-02683-7_75
McCarthy, J. (2007). What is artificial intelligence. https://aibiology.github.io/documents/intro_to_AI.pdf
Montalvo, S., Palomo, J., & de la Orden, C. (2018). Building an educational platform using NLP: A case study in teaching finance. Journal of Universal Computer Science, 24(10), 1403-1423. https://doi.org/10.3217/jucs-024-10-1403
Nabiyev, V., & Erumit, A. K. (2022). Yapay zekanın temelleri [Fundamentals of artificial intelligence]. In V. Nabiyev, & A. K. Erumit (Eds.), Eğitimde yapay zeka kuramdan uygulamaya [Artificial intelligence in education from theory to practice]. Pegem Publishing.
NCTM. (2014). Principles to actions: Ensuring mathematical success for all. National Council of Teachers of Mathematics.
Nguyen, Q. N., Sidorova, A., & Torres, R. (2022). User interactions with chatbot interfaces vs. menu-based interfaces: An empirical study. Computers in Human Behavior, 128, 107093. https://doi.org/10.1016/j.chb.2021.107093
Nichols, J. A., Herbert Chan, H. W., & Baker, M. A. (2019). Machine learning: Applications of artificial intelligence to imaging and diagnosis. Biophysical Reviews, 11, 111-118. https://doi.org/10.1007/s12551-018-0449-9
Nimavat, K., & Champaneria, T. (2017). Chatbots: An overview types, architecture, tools and future possibilities. International Journal for Scientific Research and Development, 5(7), 1019-1024.
O’Leary, D. E. (2023). An analysis of three chatbots: BlenderBot, ChatGPT and LaMDA. Intelligent Systems in Accounting, Finance and Management, 30(1), 41-54. https://doi.org/10.1002/isaf.1531
OpenAI. (2022). Introducing ChatGPT. https://openai.com/blog/chatgpt
OpenAI. (2023). ChatGPT: Optimizing language models for dialogue. OpenAI. https://openai.com/blog/chatgpt/
Qureshi, R., Shaughnessy, D., Gill, K. A., Robinson, K. A., Li, T., & Agai, E. (2023). Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? Systematic Reviews, 12(1), 72. https://doi.org/10.1186/s13643-023-02243-z
Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI. https://openai.com/research/language-unsupervised
Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783. https://doi.org/10.3390/app13095783
Ray, S. (2019). A quick review of machine learning algorithms. In Proceedings of the 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (pp. 35-39). IEEE. https://doi.org/10.1109/COMITCon.2019.8862451
Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023). Large language models (GPT) struggle to answer multiple-choice questions about code. arXiv. https://doi.org/10.5220/0011996900003470
Short, C. E., & Short, J. C. (2023). The artificially intelligent entrepreneur: ChatGPT, prompt engineering, and entrepreneurial rhetoric creation. Journal of Business Venturing Insights, 19, e00388. https://doi.org/10.1016/j.jbvi.2023.e00388
Strzelecki, A. (2023). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments. https://doi.org/10.1080/10494820.2023.2209881
Susnjak, T. (2022). ChatGPT: The end of online exam integrity? arXiv.
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
Turing, A. M. (1950). Computing machinery and intelligence. Creative Computing, 6(1), 44-53. https://doi.org/10.1093/mind/LIX.236.433
Voskoglou, M. G., & Salem, A. B. M. (2020). Benefits and limitations of the artificial with respect to the traditional learning of mathematics. Mathematics, 8(4), 611. https://doi.org/10.3390/math8040611
Wang, J., Liu, Z., Zhao, L., Wu, Z., Ma, C., Yu, S., Dai, H., Yang, Q., Liu, Y., Zhang, S., Shi, E., Pan, Y., Zhang, T., Zhu, D., Li, X., Jiang, X., Ge, B., Yuan, Y., Shen, D., ..., & Zhang, S. (2023). Review of large vision models and visual prompt engineering. arXiv. https://doi.org/10.1016/j.metrad.2023.100047
Wardat, Y., Tashtoush, M. A., AlAli, R., & Jarrah, A. M. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. EURASIA Journal of Mathematics, Science and Technology Education, 19(7), em2286. https://doi.org/10.29333/ejmste/13272
Wei, Y., Yang, Q., Chen, J., & Hu, J. (2018). The exploration of a machine learning approach for the assessment of learning styles changes. Mechatronic Systems and Control, 46(3), 121-126. https://doi.org/10.2316/journal.201.2018.3.201-2979
Weizenbaum, J. (1966). ELIZA–A computer program for the study of natural language communication between man and machine. Commununications of the ACM, 9(1), 36-45. https://doi.org/10.1145/365153.365168
Yang, S. J., Ogata, H., Matsui, T., & Chen, N. S. (2021). Human-centered artificial intelligence in education: Seeing the invisible through the visible. Computers and Education: Artificial Intelligence, 2, 100008. https://doi.org/10.1016/j.caeai.2021.100008
Yue, P., Shangguan, B., Hu, L., Jiang, L., Zhang, C., Cao, Z., & Pan, Y. (2022). Towards a training data model for artificial intelligence in earth observation. International Journal of Geographical Information Science, 36(11), 2113-2137. https://doi.org/10.1080/13658816.2022.2087223
Zhang, K., & Aslan, A. B. (2021). AI technologies for education: Recent research & future directions. Computers and Education: Artificial Intelligence, 2, 100025. https://doi.org/10.1016/j.caeai.2021.100025

How to cite this article

APA

Korkmaz Guler, N., Dertli, Z. G., Boran, E., & Yildiz, B. (2024). An artificial intelligence application in mathematics education: Evaluating ChatGPT’s academic achievement in a mathematics exam. Pedagogical Research, 9(2), em0188. https://doi.org/10.29333/pr/14145

Vancouver

Korkmaz Guler N, Dertli ZG, Boran E, Yildiz B. An artificial intelligence application in mathematics education: Evaluating ChatGPT’s academic achievement in a mathematics exam. PEDAGOGICAL RES. 2024;9(2):em0188. https://doi.org/10.29333/pr/14145

AMA

Korkmaz Guler N, Dertli ZG, Boran E, Yildiz B. An artificial intelligence application in mathematics education: Evaluating ChatGPT’s academic achievement in a mathematics exam. PEDAGOGICAL RES. 2024;9(2), em0188. https://doi.org/10.29333/pr/14145

Chicago

Korkmaz Guler, Nezihe, Zeynep Gul Dertli, Elif Boran, and Bahadir Yildiz. "An artificial intelligence application in mathematics education: Evaluating ChatGPT’s academic achievement in a mathematics exam". Pedagogical Research 2024 9 no. 2 (2024): em0188. https://doi.org/10.29333/pr/14145

Harvard

Korkmaz Guler, N., Dertli, Z. G., Boran, E., and Yildiz, B. (2024). An artificial intelligence application in mathematics education: Evaluating ChatGPT’s academic achievement in a mathematics exam. Pedagogical Research, 9(2), em0188. https://doi.org/10.29333/pr/14145

MLA

Korkmaz Guler, Nezihe et al. "An artificial intelligence application in mathematics education: Evaluating ChatGPT’s academic achievement in a mathematics exam". Pedagogical Research, vol. 9, no. 2, 2024, em0188. https://doi.org/10.29333/pr/14145