Responsible unit: Office of the President

Contact: Anna-Karin Björling och Camilla Norrbin

Introduction

The university’s quality policy states that “the quality system includes processes where research and education are regularly reviewed and monitored, in order to ensure quality and development of the core operations.” The purpose of this governing document is to describe the quality-assuring activity of programme review within the quality system for education and the regulations that govern the work with these reviews at Stockholm University.

Background

The university’s quality system for education can be described as comprising six areas: 1) organisation and governance, 2) establishment, revision and discontinuation, 3) planning, implementation and follow-up, 4) admission and degrees, 5) employment and continuing professional development, and 6) student support and information. Each area includes the conditions, processes and activities that contribute to ensuring and developing the quality of courses and study programme, such as decision-making procedures and delegation of authority, course syllabi, establishment of courses and study programmes, course evaluations, constructive alignment, well-conceived instruction, awarding of degrees, peer review for teaching positions and study and career guidance. Quality assurance is conducted by means of activities such as programme reviews, quality dialogues, focus reviews and a system of course evaluations and course reports.

Programme reviews

The programme reviews are a quality assurance activity spanning several of the six areas referred to above and aiming to generate the regular and systematic knowledge needed to ensure and develop the quality of the university’s courses and study programmes. The term “courses and study programmes” refers to both degree and non-degree programmes. Courses and study programmes thus include courses and study programmes that form part of main fields of study and vocational education and training, freestanding courses, contract education, and courses and study programmes in third-cycle subject areas. However, non-credit-bearing courses and study programmes are not included. The reviews do not focus primarily on subject-specific quality, but on conditions, procedures and processes regarding the planning, implementation and follow-up of courses and study programmes. For the most part, subject-specific quality is inspected in the context of external review, where applicable. External review thus ensures that the programme reviews as an activity in the system also develop the subject quality of the courses and study programmes.

A programme review comprises the following six steps:

  1. Start-up,
  2. work on the programme report and other documentation,
  3. review of the programme report and other documentation,
  4. preparation of decision,
  5. decision, and
  6. follow-up of decision.

The decision (step 5 above) may include one or more of the following:

  • No recommendations and/pr necessary measures,
  • recommendations,
  • necessary actions,
  • discontinuation of courses and study programmes, and
  •  external review.

Depending on the type of measure, the follow-up varies and can take place at different points in the review cycle.

Summaries of the results of the programme reviews form one of the documents in the annual quality report and the subsequent quality dialogue between the two science academic areas and the President.

The review process can be illustrated as shown in Figure 1 below. (The figure can only be viewed in the attached PDF-file).

The results of the reviews are published on the university’s website. This includes the review report and the decision of the boards of the science academic areas. If the necessary measures form part of a decision, the scientific area board’s decision after submission of the action report must also be published. If an external review has been conducted, the review report – or relevant parts of it – and the of the scientific area board’s decision must be published.

Responsibilities of the President and the scientific area boards

The President adopts the university-wide regulations for programme reviews and the associated mandatory templates. Otherwise, the scientific area boards have the following responsibilities.

The scientific area boards shall establish a procedure document within each science academic area that includes the planning, implementation and follow-up of the programme reviews.

The scientific area boards shall decide on the following:

  • categorisation and grouping of courses and study programmes,
  • schedule for the review of each programme within the framework of the review cycle,
  • review panels for each new round of reviews,
  • recommendations and necessary actions for the courses and study programmes reviewed,
  • appointment of external reviewers if a decision is made to hold an external review, and
  • follow-up of necessary measures from internal and any external reviews within a specified timeframe.

The scientific area board is also responsible for providing feedback to the department responsible for the programme on the decisions and results relating to the programme review, and for publishing the results of the reviews.

The schedule for the reviews must be communicated to the Office of the President and Student Services no later than three months before a review cycle commences. Similarly, any revisions to the schedules must be communicated so that planning for register maintenance, for example, can take place well in advance of the review starting.

If the set of courses or study programme is delivered in cooperation between departments in both science academic areas, as is the case for teacher education, the review must be conducted in consultation between the areas.

Rebus responsibilities – development and evaluation

The President’s Advisory Board for University-Wide Educational Issues and Systems for Quality Assurance (Rektors beredning för universitetsgemensamma utbildningsfrågor och system för kvalitetssäkring, or Rebus) is responsible for, if necessary, developing, revising and making proposals to the President regarding adoption of the establishment of the mandatory templates used in programme reviews. Rebus is also responsible for revising the support material available.1

Rebus is also responsible for continuously evaluating the methodology, conditions and process for programme review and, if necessary, making proposals to the President regarding revision and adoption of this regulatory document.

An evaluation of the programme review process will be conducted by the time of the next quality cycle, including a random review of degree projects from a selection of the courses and study programmes reviewed.

Regulations for programme reviews

Lenght of the review cycle

All courses and study programmes must be reviewed within a review cycle. The duration of the review cycle is six years.

Postponment, exemption or limited programme review

During a review cycle, the scientific area board may decide to postpone or exempt a specific course or study programme from programme review in instances where the programme is included in the Swedish Higher Education Authority’s (UKÄ’s) programme evaluation, or if the programme is intended to undergo accreditation. The scientific area board may also decide on a limited programme review of a degree programme for the same reasons. In the case of a limited programme review, the programme is reviewed against a selection of the quality criteria described below, which are not covered by UKÄ’s programme evaluation, for example.

For practical reasons, other programmes may also require adaptation of the process and quality criteria, e.g. “odd courses”, introductory courses, contract education, etc. The forms of such adapted programme review are adopted by the scientific area board and reported to Rebus.

Courses and study programmes that are part of a collaboration with other national or international higher education institutions are reviewed as agreed between the institutions.

Quality criteria

All study programmes and set of related courses must be assessed on the basis of quality criteria which are designed primarily using the framework of the Higher Education Act (1992:1434), the Higher Education Ordinance (1993:100) and the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). Each programme under review must be assessed on the basis of all quality criteria, with the exception of non-degree programmes, Supplementary training programme for foreign teachers (Utländska lärares vidareutbildning, ULV) and Supplementary training programme for teachers
(Vidareutbildning av lärare, VAL). A selection of the criteria must be used in these cases.

The quality criteria, which are also included in the templates for the programme reports and review reports (under headings numbered in the same way as the abbreviations in the table below, i.e. 1.2, 1.3, etc.), are as follows:

First and second-cycle levels (GN/AN)

Third-cycle level (FN)

K 1.2 GN/AN There are procedures for the follow-up and analysis of student completion rates, and measures are taken when necessary.

K 1.3 FN Measures are taken and support is given, if necessary, to create good conditions for the doctoral student to complete the programme within the planned period of study.

K 1.3 GN/AN The teachers (and other teaching staff) and their combined scientific, professional, and pedagogical competence are adequate and in line with the volume, content, and implementation of the evaluated programme or set of courses, in both the short and long term.

K 1.4 FN The teachers/supervisors and their combined scientific and pedagogical competence are adequate and are in line with the volume, content and implementation of the evaluated programme in both the short and long term.

K 1.4 GN/AN The programme or set of courses are characterised by a close connection between research and education.

K 1.5 FN The research environment is of such a quality and scope that the programme can be conducted at a high scientific level and with a wide range of subjects.

K 1.5 GN/AN Administration, infrastructure and student support for the programme or set of courses are appropriate and promote quality development.

K 1.6 FN Administration, infrastructure and doctoral student support for the programme are appropriate and promote quality development.

K 1.6 GN/AN There are appropriate and systematic procedures and processes that ensure the quality of the degree theses.

 K 1.7 FN There are appropriate and systematic procedures and processes concerning the announcement of positions, so that recruitment is transparent and broad, and the admission process takes place with care and in a legally secure manner.

 

K 1.8 FN There are appropriate and systematic procedures and processes for appointing and changing supervisors.

K 2.1 GN/AN The design, implementation and examination of the programme or set of courses ensures that students are given the opportunity to achieve the qualitative targets.

K 2.1 FN The design, implementation and examination of the programme ensure that doctoral students are given the opportunity to achieve the qualitative targets.

 

K 2.2 FN The individual study plan is used in an appropriate and systematic way to design, plan, and follow-up the individual doctoral student's education in relation to the qualitative targets.

K 3.1 GN/AN The programme or set of courses are continuously followed-up, the results are fed back to relevant parties, and based on the results, the measures required to improve and develop the programmes are taken.

K 3.1 FN The programme is continuously followed-up, the results are fed back to relevant parties, and based on the results, the measures required to improve and develop the programmes are taken.

 

K 3.2 FN There are appropriate and systematic procedures and processes that ensure the quality of doctoral dissertations and licentiate theses.

K 3.2 GN/AN There are appropriate and systematic procedures and processes where students are given the opportunity to exercise influence over their education and study environment, both as individuals and collectively.

K 3.3 FN There are appropriate and systematic procedures and processes where doctoral students are given the opportunity to exercise influence over their education and study environment, both as individuals and collectively.

K 3.3 GN/AN The work with gender equality in the programme or set of courses is relevant and appropriate.

K 3.4 FN The work with gender equality in the programme is relevant and appropriate.

K 3.4 GN/AN The work with widening participation is relevant and appropriate.

 

K 3.5 GN/AN There are processes which ensure that the programme or set of courses are useful and prepare students to assess changes in their work-lives.

K 3.5 FN There are processes which ensure that the programme is useful and prepares doctoral students to assess changes in their work-lives.

 

Student representation

The student union shall be offered the opportunity to appoint student representatives (i.e. students and doctoral students) for all stages of the programme reviews. Student representatives participate in the review process on the same terms as other participants. The number of student representatives and replacements during the review process is set out in “Regulations for student influence”. Students must also be compensated for their efforts in accordance with “Regulations for student influence”, and doctoral students must be compensated in accordance with “Regulations for third cycle education and examination”.

Start-up and work with documentation

A start-up meeting shall take place before each review round.

The programme report must be written in the designated template for first and second cycle programmes and third cycle programmes.

If the programme spans several departments, the report must be approved by all the departments involved.

The Student Union must be offered the opportunity to appoint student representatives to work on the programme report. These representatives may benefit from studying at the department responsible for the programme.

The template for the first and second cycle programme report includes a teacher/supervisor table, a qualitative target matrix and compiled statistical data.

The template for the third cycle programme reports also includes a teacher/supervisor table, a general study syllabus and a selection of individual study plans, as well as compiled statistical data.

Compiled statistical data is provided by the relevant scientific area office, while the teacher/supervisor table and qualitative target matrix (qualitative target matrix for professional degrees is provided by the relevant scientific area office) and general study plan are published on the staff website. Departments select individual study plans as instructed by the scientific area office, and these are attached to the programme report.

Other documentation needed to assess whether the study programme or set of courses meets the quality criteria may be included in the programme review upon request. Examples of what may constitute other documentation can be found in the programme report templates.

The department board or head of department responsible must submit the final version of the programme report and any other documentation to the scientific area office.

Peer review of the documentation

The programme report and other documentation must be reviewed by a review panel comprising academic members appointed by the scientific area board and one or more student representatives appointed by the Student Union in accordance with “Regulations for student influence”. The academic members must hold doctoral degrees and be employed for an indefinite period, or hold a position as an associate senior lecturer at Stockholm University.

Each review group must comprise members from several faculties/sections, if possible, from both science academic areas.

Each review panel must have a chair whose role, it is to coordinate the work of the review panel, as well as acting as a reviewer. Office support must be available to the review groups and, if necessary, assist the reviewers with information on national and local regulatory frameworks related to courses and study programmes

Dialogue

During the review period, a dialogue must be held between the review panel and course or programme coordinators. Student representatives from the review panel and student representatives who participated in work on the programme report at the department responsible for the programme must be offered the opportunity to participate. One of the aims of the dialogue is to clarify any ambiguities in the documentation.

Review report and proposals

The review panel is jointly responsible for reporting its work in a review report. The template for the review report reflects the content of the programme report template.

Working on the basis of the quality criteria, the panel may propose recommendations and/or necessary actions to improve the quality and development of the programme. Recommendations and necessary actions must be clearly linked to problems identified. There is thus a difference compared to more general proposals and advice, which may be included in the body of the review report but not in the actual proposed recommendations and/or necessary actions. The review must not result in an evaluative final assessment of the entire set or courses or study programme.

The review panel may propose an external review if it finds that a further review is necessary.

Preparation before decision

Before the review panel finalises its work, the programme coordinators and student representatives concerned must be given the opportunity to comment on any factual errors and ambiguities in the review report.

The relevant education committee, or other form of preparatory group, must consider the review panel’s proposals and deliberations and, after calibration, submit a proposal for a decision to the scientific area board.

The programme coordinators and the student representatives concerned must be notified of the proposed decision when the proposed decision is sent to the scientific area board.

Decisions and feedback after decisions

The scientific area board must then make a decision on the matter according to the options listed on page 4.

The scientific area board’s decision must be communicated to the programme coordinators and the members of the concerned panel.

Quality-driven dialogue

When the scientific area board has made a decision on the matter, a dialogue must take place with programme coordinators and student representatives. The purpose of this dialogue is to drive quality.

Follow-up after decisions

The monitoring of the recommendations adopted by the scientific area board will take place at the latest when it is time for a new review of the programme, but the board may also decide on earlier follow-up.

A programme that has been assessed as being in need of necessary action must submit a report on this action to the scientific area board within a given time frame. The process for this work is decided by the board.

The action report is compiled by the department responsible for the programme in accordance with the science academic areas procedure document and the relevant template.

If the follow-up indicates persistent quality deficiencies, the scientific area board must decide on further handling of the development work.  

External review

The scientific area board may decide that an external review is to be conducted. This decision can be made at any time during the review process.

The scientific area board also decides on:

  • which subject expert reviewers are to be included in the external assessment panel. There must always be at least two subject experts, who have publicly defended their doctoral theses and who are employed at other higher education institutions,
  • the time available to the assessment panel for the review,
  • the forms of external review and documentation included in it. If the external review relates to education at first or second cycle level, degree projects from the programme in question must be included as documentation, if possible. See also Review of degree projects below,
  • any recommendations and/or necessary measures while the external review is being conducted, and
  • forms of follow-up and feedback to the department responsible for the programme and any recommendations and/or necessary measures once the external review is completed.

The results of the review must be published on the university’s quality assurance website, with the exception of the subject experts’ assessments of degree projects where these are used as a basis.

Review of degree projects

The subject expert reviewers must individually assess at least five-degree projects (or, if fewer, as many as are available) from the study programme in question for a maximum of three years back in time. Each degree project must be assessed by at least two reviewers.

The review must be based on five assessment aspects: research basis; problem formulation; method, implementation and outcome; discussion and conclusions; and formulation, language and formalities.

The scientific area board has the opportunity:

  • to clarify or add to one or more of these aspects, e.g. in the light of the nature of the subject or discipline. This must then be indicated in the decision on external review made by the scientific area board, and
  • to select an appropriate model for the random selection of degree projects for the programme in question.

The individual review must result in an opinion, according to a predetermined five-point scale and reasons/comments per aspect for each degree project. Each reviewer must also provide an overall opinion per aspect based on all degree projects reviewed. The reviewers must also provide an overall assessment and reflection on the degree projects reviewed. In other words, the review must not result in a collective opinion per degree project.

More detailed information on external review, e.g. on how degree projects are to be reviewed, guiding questions for the review and examples of relationships between assessment aspects and qualitative targets, can be found in the preparatory work (“Metod för pilottestning av extern granskning i samband med utbildningsgranskningar” [Method for pilot testing of external review in connection with programme reviews] of 4 November 2019, available on the quality assurance website).

Quality dialogue and managing of the results of the
programme reviews

The results of the programme reviews must constitute one of the documents in the quality report that the scientific area board submit annually to the President prior to the quality dialogues that take place between the scientific area board in question and the President. Good examples must be specifically highlighted in the report in order to facilitate further dissemination of these within the university to heads of department and directors of studies, for example. The quality report template is available on the quality assurance website.

The purpose of the quality dialogue is to create a platform for the exchange of information between the President, Vice-President and scientific area board on quality issues, and to follow up on the scientific academic areas’ quality enhancement work. The results of the quality dialogues will be used as a basis within the framework of the university’s regular operational planning, i.e. in the university’s overall strategies and in the scientific area board’ and administration’s respective action plans, or in ongoing activities.