Stockholm university
Department of Education

Hello GPT! Goodbye home examination?

In a new study university teachers blindly assessed student and ChatGPT-written responses to home examination questions. The findings show that ChatGPT texts achieved a high passing grade and that teachers were more critical when grading student-written texts.

Chat Gpt on a mobila phone
Foto: Martin of Sweden

The study’s aim was to examine ChatGPT’s and similar AI chatbot's impact on university teachers’ assessment practices and their potential impact on higher education. University teachers from four different departments in humanities and social sciences participated in the test. Additionally, interviews were conducted with the same teachers examining their reflections about the quality of the texts they assessed.

 

 

Findings in short

 

Participants’ assessment of AI chatbot responses

  • ChatGPT responses achieved a passing grade between 37.5% (education) and 85.7% (philosophy).
  • The participants highlighted the quality of language and texts were perceived as creative and innovative.
  • The weaknesses of ChatGPT texts were argumentation strategy, use of references, relevance to the content of the course and the course literature.

Participants’ assessment of student responses

  • Patterns of downgrading were identified, suggesting a more critical grading of student-written texts.
  • The participants were more critical towards student responses regarding structural and aesthetical flaws.
  • Several of the participants suspected that some of the student texts assessed were bot-written.
 

Conclusions

Our study shows that ChatGPT may have an impact on university teachers’ assessment practices in amplifying their criticality and suspicion and challenging their trust in students’ texts.

We also identified that participants were prone to assess differently than they usually did because they were aware of the potential use of ChatGPT in the texts presented to them.

The presence of AI chatbots may also disrupt teachers’ assessment practices and question their inherent trust in the students writing their home examinations by themselves.

 

 

Link to the study

Hello GPT! Goodbye home examination? An exploratory study of AI chatbots impact on university teachers’ assessment practices

 

The researchers

Alexandra Farazouli, PhD student, Department of Education

Teresa Cerratto-Pargman, professor, Department of Computer and Systems Sciences

Klara Bolander-Laksov, professor, Department of Education

Cormac McGrath, Senior lecturer, Department of Education

Theme
Digitalization and AI - Theme page
Research subjects
Research in Higher Education Technology Enhanced Learning
Research group

Last updated: September 7, 2023

Source: IPD

eventNewsArticle

standard-article

false

{
  "dimensions": [
    {
      "id": "department.categorydimension.subject",
      "name": "Global categories",
      "enumerable": true,
      "entities": [],
      "localizations": {}
    },
    {
      "id": "department.categorydimension.tag.Keywords",
      "name": "Keywords",
      "enumerable": false,
      "entities": [],
      "localizations": {}
    },
    {
      "id": "department.categorydimension.tag.Person",
      "name": "Person",
      "enumerable": false,
      "entities": [],
      "localizations": {}
    },
    {
      "id": "department.categorydimension.tag.Tag",
      "name": "Tag",
      "enumerable": false,
      "entities": [],
      "localizations": {}
    },
    {
      "id": "webb2021.categorydimension.Category",
      "name": "Globala kategorier Nyheter (Webb 2021)",
      "enumerable": true,
      "entities": [
        {
          "id": "webb2021.categorydimension.Category.research.news",
          "name": "Research",
          "entities": [],
          "attributes": [],
          "childrenOmitted": false,
          "localizations": {}
        }
      ],
      "localizations": {}
    },
    {
      "id": "webb2021.categorydimension.Label",
      "name": "Tema (Webb 2021)",
      "enumerable": true,
      "entities": [],
      "localizations": {}
    },
    {
      "id": "webb2021.categorydimension.Label.en",
      "name": "Themes (Webb 2021)",
      "enumerable": true,
      "entities": [
        {
          "id": "webb2021.categorydimension.Label.en.CompSci",
          "name": "Digitalization and AI",
          "entities": [],
          "attributes": [],
          "childrenOmitted": false,
          "localizations": {}
        }
      ],
      "localizations": {}
    },
    {
      "id": "webb2021.categorydimension.Keyword",
      "name": "Keywords (Webb 2021)",
      "enumerable": false,
      "entities": [],
      "localizations": {}
    },
    {
      "id": "IPD.eng.lokala.kat",
      "name": "Lokala kategorier ENG IPD",
      "enumerable": true,
      "entities": [
        {
          "id": "IPD.eng.lokala.kat.nyh",
          "name": "Local news",
          "entities": [
            {
              "id": "IPD.eng.lokala.kat.nyh.fo",
              "name": "Research",
              "entities": [],
              "attributes": [],
              "childrenOmitted": false,
              "localizations": {}
            }
          ],
          "attributes": [],
          "childrenOmitted": false,
          "localizations": {}
        }
      ],
      "localizations": {}
    }
  ]
}