Evaluating Research and Scholarly Impact in Criminology and Criminal Justice in the United Kingdom and Italy: A Comparative Perspective

• Published date: 01 August 2023
• DOI: http://doi.org/10.1177/10439862231170966
• Author: Anita Lavorgna,Pamela Ugwudike,Francesca Vianello
• Date: 01 August 2023

https://doi.org/10.1177/10439862231170966

Journal of Contemporary Criminal Justice

2023, Vol. 39(3) 354 –370

© The Author(s) 2023

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/10439862231170966

journals.sagepub.com/home/ccj

Article

Evaluating Research

and Scholarly Impact in

Criminology and Criminal

Justice in the United Kingdom

and Italy: A Comparative

Perspective

Anita Lavorgna1,2, Pamela Ugwudike2,

and Francesca Vianello3

Abstract

What scholarly impact is, and how it is evaluated, vary across different countries. In

the United Kingdom, for instance, scholarly impact is mainly assessed through the

Research Excellence Framework (REF) in the context of providing—among other

things—accountability for public investment in research, demonstrating the public

benefits of research, and informing the selective allocation of research funding. In

the REF system, impact needs to show a demonstrable effect on change, or evidence

of benefits outside academia, and is formally assessed through case studies. In

Italy, there is a comparable system for evaluating research, known as Evaluation

of Research Quality, but in this latter case, the focus is on the quality of selected

research outputs as indicators of research performance. Impact is here considered

with reference to the so-called third mission (which includes activities aimed at the

valorization of research, and activities that have positive spillovers into society at

large) and is evaluated separately. Our contribution aims at critically analyzing the

commonalities and differences of these two systems when it comes to evaluating

research in Criminology and Criminal Justice, considering some of the benefits and

potential pitfalls of research evaluation in both regions, and discussing how these

disciplines are framed and delimited differently in the two countries considered.

1University of Bologna, Italy

2University of Southampton, UK

3University of Padova, Italy

Corresponding Author:

Anita Lavorgna, Department of Political and Social Sciences, University of Bologna, Via Zamboni, 33,

Bologna, 40126, Italy.

Email: anita.lavorgna@unibo.it

1170966CCJXXX10.1177/10439862231170966Journal of Contemporary Criminal JusticeLavorgna et al.

research-article2023

Lavorgna et al. 355

Keywords

research evaluation, impact, Italy, United Kingdom, Research Excellence Framework

(REF), Evaluation of Research Quality (VQR), comparative analysis

Introduction

In a research context characterized by scarce resources, especially over the last couple

of decades, several ways to assess research quality, particularly to measure scholarly

productivity and the scientific impact of academic institutions, funding agencies, and

other parties, have been proposed. Key measures include citation analysis, at times

supplemented by other means to try and assess readership data, and each measure has

its merits and shortcomings (e.g., Bollen et al., 2005; Moed, 2005; Mohammadi &

Thelwall, 2014; Priem & Hemminger, 2010; Rowlands & Nicholas, 2007; Schloegl &

Gorraiz, 2011). As we will see, the measures are not neutral, but they are—at least

implicitly—aligned with a certain approach to framing and understanding academic

research, and have both direct and indirect repercussions for how certain research

endeavors are prioritized or even implemented.

This contribution focuses on the two main nation-wide systems used in the United

Kingdom and Italy to measure or evaluate research and assess scholarly impact beyond

academia. Both countries have similar overall and student population sizes (Rebora &

Turri, 2013) and the systems used to assess impact are the Research Excellence

Framework (hereafter, REF) in the United Kingdom and the Evaluation of Research

Quality (Valutazione della Qualità della Ricerca, hereafter, VQR) in Italy. After pre-

senting a descriptive overview of both systems, this article will draw on the relevant

literature to critically discuss some systemic limitations affecting them. The chapter

will discuss the specific deficiencies of the current systems, with a focus on their

application in the field of Criminology and Criminal Justice (hereafter, CCJ).

Before moving to the core of this contribution, a brief definitory premise is neces-

sary. CCJ is a broad field of teaching and research. Its definition and disciplinary divi-

sions vary across diverse countries according to their unique theoretical traditions and

linguistic preferences. In the United Kingdom, for instance, CCJ is more clearly

detectable as a stand-alone discipline in research publications and teaching programs

(for a relatively recent overview, see Harris et al., 2019). It is mostly recognized as an

interdisciplinary and evolving field that brings together theories, principles, and meth-

ods from law, psychology, economics as well as sociology and other social sciences. It

finds its core in its focus on the etiology of crime and deviance, and the operation of

criminal justice systems (e.g., Brisman et al., 2018; Downes, 1988; Garland & Sparks,

2000; Zedner, 2007). Over time, the boundaries of the discipline have been stretched,

to encompass relative new areas such as the study of social harms. The discipline has

also become more open to inputs from disciplines from outside the social sciences

(consider, for instance, the computer and data sciences), giving rise to the nascent field

of digital criminology. In Italy, however, CCJ stems from legal and psychological

traditions. At the time of writing, CCJ in most universities tends to be academically