A Comparative Study of GPT3.5, GPT4, and Bard- Gemini-Gemini in Addressing Object-Oriented Programming Tasks | [International Journal of Communication Networks and Information Security • 2024]

Author(s):

1. Suvarnalata Hiremath: Department of Computer Science and Engineering, Visvesvaraya Technological University (VTU), Belagavi, Karnataka,India

2. C M Tavade: Department of Electronics and Communication Engineering, SGBIT, Belagavi. Visvesvaraya Technological University (VTU), Belagavi, Karnataka, 590018, India

Abstract:

Large Languages Model (LLMs) having shown great potentiality as assistive tools for pupils engaged in programming assignments. Nevertheless, the intricate nature of object-oriented programming (OOP), with its multifaceted requirements for entity identification, relationship establishment, and responsibility delineation, remains a challenge for these models. While LLMs have shown proficiency in introductory programming tasks, their performance in the context of OOP has been under-researched. Current studies targeted towards addressing these gaps by evaluating the efficacy of three prominent LLMs-GPT3.5, GPT4, and Bard-Gemini-in solving current world's OOPs works utilized inside education-based setting. Answers generated by these models were subsequently validated using an Automatic Assessment Tool (AAT). Results indicate thatwhile the models were frequently capable of producing functional solutions, they often failed to adhere to established OOP best practices. Among the evaluated models, GPT4 exhibited the highest proficiency, followed by GPT3.5, with Bard-Gemini-Gemini demonstrating the least proficiency. Based on these findings, the researcher advocates for a heightened emphasis on code quality when utilizing LLMs for educational purposes and suggests integrating Large Languages Model with AATs like a potential channels towards pedagogical advancement. The study concludes by highlighting the promising capabilities of GPT4 while emphasizing the necessity for continued supervision in the deployment of LLMs within OOP education.

Page(s): 849-859

DOI: DOI not available

Published: Journal: International Journal of Communication Networks and Information Security, Volume: 16, Issue: S1, Year: 2024

Keywords:

GPT35 , ObjectOriented Programming Tasks , Bard GeminiGemini

References:

[1] Anonymous .2023 .This is the anonymized version to support peer review. , : .

[2] Barredo Alejandro,Arrieta Alejandro,Bennetot Adrien .2020 .Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion 58, : 82-115.

[3] Brown Tom,Jared D Kaplan Amanda,Askell Amanda .2020 .Language Models are Few-Shot Learners. Advances in neural information processing systems, 33 : 1877-1901.

[4] Bubeck Sébastien,Lee Peter,Li Yuanzhi,Lundberg Scott .2023 .Early experiments with GPT4. arXiv preprint arXiv:2303.12712, : .

[5] Cai Yuzhe,Wang You,Xia Yan .2023 .Low-code LLM: Visual Programming over LLMs. , : .

[6] .2023 .. arXiv preprint arXiv:2304.08103, : .

[7] Chen Mark,Brockman Greg .2021 .. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, : .

[8] Pereira Bruno,Alves Pedro,Alves Pedro .2023 .How GPT3.5, GPT4 and Bard-Geminihandled an Object Oriented Programming Assignment - Full Interaction Logs. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science, 1(Turku) : 61-67.

[9] Pereira Bruno,Cipriano Bruno,Alves Pedro .2022 .Drop Project: An automatic assessment tool for programming assignments. SoftwareX, 18 : 101079.

[10] Daun Marian,Brings Jennifer .2023 .How ChatGPT Will Change Software Engineering Education. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, : 110-116.

[11] Denny Paul,Brent N Reeves Paul .2023 .Promptly: Using Prompt Problemsto Teach Learners How to Effectively Utilize AI Code Generators. arXiv preprint arXiv:2307.16364, : .

[12] Destefanis Giuseppe,Ortu Marco .2023 .A Preliminary Analysison the Code Generation Capabilities of GPT3.5 and Bard-Gemini AI Models for Java Functions. arXiv preprint arXiv:2305.09402, : .

[13] .2022 .The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing EducationConference, : 10-19.

[14] Finnie-Ansley James .2023 .My AI Wants to Know if This Will Be on the Exam: Testing OpenAI's Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference, 97 : 104.

[15] Ivanov Roman .2023 .. , : .

[16] Hieke Keuning,Heeren Bastiaan .2023 .A Systematic Mapping Study of Code Quality in Education. In Proceedings of the 2023 Conference on Innovationand Technology in Computer Science Education V. 1, : 5-11.

[17] Lau and Philip J Guo Sam .2023 .From" Ban It TillWe Understand It" to" Resistanceis Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. , : .

[18] Leinonen Juho,Tran Andrew,Hellas Arto .2023 .Comparing Code Explanations Created by Students and Large Language Models. arXiv preprint arXiv:2304.03938, : .

[19] Liffiton Mark,Denny Paul .2023 .CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. arXiv preprint arXiv:2308.06921, : .

[20] Julie E,Mills E .2003 .Engineering education-Is problem basedor projectbased learning the answer. Australasian journal of engineering education 3, (2) : 2-16.

[21] Elena N,Naumova N .2023 .A mistake-find exercise: a teacher's tool to engage with information innovations, ChatGPT, and their analogs. Journal of Public Health Policy 44, (2) : 173-178.

[22] OpenAI .2023 .GPT4 Technical Report. , : 2303.

[23] Lieh Eng,Ouh Eng,Wlodkowski Swavek .2023 .ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course. arXiv preprint arXiv:2305.13680, : .

[24] Parsons Dale,Haden Patricia .2006 .Parson's Programming Puzzles: A Fun and Effective Learning Tool for First Programming Courses. In Proceedings of the 8th Australasian Conference on Computing Education-, 52 : 157-163.

[25] Brent Reeves,Leinonen Juho .2023 .Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, : 299-305.

[26] Jaromir Savelka,An Marshall,Bogart Chris,Sakr Majd .2023 .Thrilled by Your Progress! Large Language Models (GPT4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. arXiv preprint arXiv:2306.10073, : .

[27] Jaromir Savelka,Sakr Majd .2023 .Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses? arXiv preprint. , 2303 : .

[28] Myeong-Hee Shin .2018 .Effects of Project-Based Learning on Students' Motivation and SelfEfficacy. English Teaching, 73(1) : 95-114.

[29] Sundar Pichai .2023 .google-ai-search-updates/. [Online; last accessed 10-August-2023]., : .

[30] Romal Thoppilan,Du Yu .2022 .Lamda: Language Models for Dialog Applications. arXiv preprint arXiv:2201.08239, : .

[31] Wegner Peter .1990 .Concepts and Paradigms of Object-Oriented Programming. ACM Sigplan Oops Messenger, 1(1) : 7-87.

[32] Wermelinger Michel .2023 .Using GitHub Copilot to Solve Simple Programming Problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, : 172-178.

[33] Frank F Xu,Alon Uri .2022 .A Systematic Evaluation of Large Language Models of Code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, : 1-10.

Citations

Downloads

Views