Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4,
  and Human Tutors
Generative AI and large language models hold great promise in enhancing
computing education by powering next-generation educational technologies for
introductory programming. Recent works have studied these models for different
scenarios relevant to programming education; however, these works are limited
for several reasons, as they typically consider already outdated models or only
specific scenario(s). Consequently, there is a lack of a systematic study that
benchmarks state-of-the-art models for a comprehensive set of programming
education scenarios. In our work, we systematically evaluate two models,
ChatGPT (based on GPT-3.5) and GPT-4, and compare their performance with human
tutors for a variety of scenarios. We evaluate using five introductory Python
programming problems and real-world buggy programs from an online platform, and
assess performance using expert-based annotations. Our results show that GPT-4
drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human
tutors' performance for several scenarios. These results also highlight
settings where GPT-4 still struggles, providing exciting future directions on
developing techniques to improve the performance of these models.
649e2f2c27145c446312a322
Tung Phung
claimed_verified
2023-07-03T11:47:13.186Z
64a27d5bcd362b2c08cdf5d1
/avatars/f3eda4e7811c15c326a38dfa18b534c2.svg
Minh Tung Phung
user
tungmphung
649e2f2c27145c446312a323
Victor-Alexandru Pădurean
claimed_verified
2023-07-03T07:27:33.716Z
649fe1259b6e48cd6b4d8615
https://cdn-avatars.huggingface.co/v1/production/uploads/649fe1259b6e48cd6b4d8615/Fz89v0I7EyxKKsRe4rwRt.jpeg
Victor Padurean
user
vicondrus
649e2f2c27145c446312a324
José Cambronero
claimed_verified
2023-07-03T07:27:45.201Z
649f4bfbb469b6b0b402b082
/avatars/25eefad619629fefb85334e14cf9bd47.svg
Jose Cambronero
user
josepablocam
649e2f2c27145c446312a325
Sumit Gulwani
649e2f2c27145c446312a326
Tobias Kohn
649e2f2c27145c446312a327
Rupak Majumdar
649e2f2c27145c446312a328
Adish Singla
claimed_verified
2023-07-03T07:27:39.952Z
649f535d9567d0d9fb75522f
/avatars/161382abfa73ee354d05eee9b468e058.svg
Adish Singla
user
adishsingla
649e2f2c27145c446312a329
Gustavo Soares
extracted_pending
2023-06-30T01:26:04.541Z
63a605226fb8987e8bb97d3b
/avatars/c6c8bc37f12d0593462099f2ea5cfe73.svg
Gustavo Soares
user
gustavosoares
Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4,
  and Human Tutors
Generative AI and large language models hold great promise in enhancing
computing education by powering next-generation educational technologies for
introductory programming. Recent works have studied these models for different
scenarios relevant to programming education; however, these works are limited
for several reasons, as they typically consider already outdated models or only
specific scenario(s). Consequently, there is a lack of a systematic study that
benchmarks state-of-the-art models for a comprehensive set of programming
education scenarios. In our work, we systematically evaluate two models,
ChatGPT (based on GPT-3.5) and GPT-4, and compare their performance with human
tutors for a variety of scenarios. We evaluate using five introductory Python
programming problems and real-world buggy programs from an online platform, and
assess performance using expert-based annotations. Our results show that GPT-4
drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human
tutors' performance for several scenarios. These results also highlight
settings where GPT-4 still struggles, providing exciting future directions on
developing techniques to improve the performance of these models.
Systematic evaluation of ChatGPT and GPT-4 in educational scenarios demonstrates GPT-4's superior performance over ChatGPT and identifies areas needing improvement.
GPT-3.5 GPT-4 ChatGPT introductory Python programming expert-based annotations