09.11 Cooperating on quality of higher education – the measuring and decision-making practices of Dutch accreditation panels

Aim of the project

This project investigates how stakeholders of Dutch higher education cooperate in assessing the quality of higher education programmes. In the Dutch accreditation system, such assessments are undertaken by panels of independent peers. These ‘accreditation panels’ are expected to measure and decide about the quality of programmes in accordance with the rules, values and goals of the accreditation system’s assessment framework. This project examines the alignment of the goals and values of the accreditation system, its assessment framework, and the measuring and decision-making practices of accreditation panels.

Theoretical background

Evaluating the quality of higher education involves the application of social scientific methods (Wanzer, 2021). In the case of Dutch accreditation reviews, these measurements are intersubjective, involving panels of independent peers; and they are multi-stake, aimed at weighing and reconciling the different perspectives and interests of higher education’s primary stakeholders. To this end, a specific measurement practice has been developed since the 1980s, culminating in a national assessment framework that was launched in 2003 (current version: NVAO Nederland, 2018).

The Dutch assessment framework makes explicit the accreditation system’s values and goals, which are both epistemic and non-epistemic in nature. For instance, assessments must be reliable, do equal justice to the panellists’ perspectives, but also place minimal administrative burden on the programme. This evaluation framework and practice can be shown to meet three important requirements of social-scientific measurements (Cartwright & Runhardt, 2014). That is: the concept of quality of higher education is formally characterised (into ‘NVAO standards’), represented (e.g. a programme ‘meets or does not meet generic quality’) and detailed into measurement procedures (e.g. ‘the panel must reach a unanimous or majority collective opinion’).

To what extent should the resulting measurement practice be expected to meet the epistemic and non-epistemic goals of the Dutch accreditation system? This project combines a normative analysis of the assessment framework’s design, with a qualitative study into the work of accreditation panels. Within the framework’s limits –or possibly transgressing them– how do accreditation panels work together and reach their collective verdict? To what extent can and do such collective (intersubjective, multi-stake) verdicts align with the accreditation system’s values and goals? And ultimately, to what extent could this measurement practice promote the sustained cooperation of all stakeholders of Dutch higher education involved in its quality assurance?

This study will build upon studies of peer review in academia (Forsberg et al., 2022; Lamont, 2009; Lee et al., 2013). Conceptualising evaluation of higher education as a measurement practice, the study will refer to philosophical studies into the construction of measurement instruments, especially intersubjective measurement practices, and with a special interest for their value- and theory-ladenness (Chang, 2008; Galison, 1987; Tal, 2016). Finally, building on judgment aggregation theory, the study will look at how this framework’s design and ensuing practice, impacts the panel’s opinion formation and judgment aggregation process (Dietrich & List, 2005; List, 2006).

Research design

The project combines qualitative research with philosophical enquiry. The philosophical enquiry is informed by –and developed in tandem with– a qualitative study into the collective measurement and decision-making processes by Dutch accreditation panels. The qualitative study is divided into two partial studies. The first partial study will involve in-depth interviews with accreditation panel chairpersons and secretaries, to explore how accreditation panels collectively assess and decide about a programme’s quality. The second partial study will involve (participatory) observations of accreditation reviews during site visits, to study in detail the interactions between panellists as they deliberate towards a collective judgment.

In addition, there will be limited (historical/archival) research into the development of the Dutch system of external quality assurance since the 1980s. This research is intended to contextualize the past and present involvement of different stakeholders in the Dutch accreditation system, and the resulting consensus regarding the system’s (epistemic and non-epistemic) purposes.

Valorization

This project’s practical and societal relevance is periodically monitored by an advisory group comprised of representatives of the following stakeholders of the Dutch accreditation system:

• Dutch-Flemish accreditation organisation NVAO.
• The Dutch Inspectorate of Education.
• Assessment agencies Academion, AeQui, Hobéon and NQA.
• Several Dutch research universities and universities of applied sciences.
• Student organizations ISO and LSVb.

In consultation with this advisory group, one of the ultimate aims of this project is to provide recommendations towards improving the current accreditation system. By analyzing the alignment of the accreditation system’s values and goals, the assessment framework’s design, and the resulting evaluation and decision-making processes by accreditation panels, this project hopes to contribute to the sustainable cooperation of stakeholders of the quality assurance of Dutch higher education.