Hello GPT! Goodbye home examination?
In a new study university teachers blindly assessed student and ChatGPT-written responses to home examination questions. The findings show that ChatGPT texts achieved a high passing grade and that teachers were more critical when grading student-written texts.
The study’s aim was to examine ChatGPT’s and similar AI chatbot's impact on university teachers’ assessment practices and their potential impact on higher education. University teachers from four different departments in humanities and social sciences participated in the test. Additionally, interviews were conducted with the same teachers examining their reflections about the quality of the texts they assessed.
Findings in short
Participants’ assessment of AI chatbot responses
- ChatGPT responses achieved a passing grade between 37.5% (education) and 85.7% (philosophy).
- The participants highlighted the quality of language and texts were perceived as creative and innovative.
- The weaknesses of ChatGPT texts were argumentation strategy, use of references, relevance to the content of the course and the course literature.
Participants’ assessment of student responses
- Patterns of downgrading were identified, suggesting a more critical grading of student-written texts.
- The participants were more critical towards student responses regarding structural and aesthetical flaws.
- Several of the participants suspected that some of the student texts assessed were bot-written.
Conclusions
Our study shows that ChatGPT may have an impact on university teachers’ assessment practices in amplifying their criticality and suspicion and challenging their trust in students’ texts.
We also identified that participants were prone to assess differently than they usually did because they were aware of the potential use of ChatGPT in the texts presented to them.
The presence of AI chatbots may also disrupt teachers’ assessment practices and question their inherent trust in the students writing their home examinations by themselves.
Link to the study
The researchers
Alexandra Farazouli, PhD student, Department of Education
Teresa Cerratto-Pargman, professor, Department of Computer and Systems Sciences
Klara Bolander-Laksov, professor, Department of Education
Cormac McGrath, Senior lecturer, Department of Education
Last updated: September 7, 2023
Source: IPD