School Library Research: Publication Bias and the File Drawer Effect?

Author:Stefl-Mabry, Joette


The need for empirical investigation into the effect of school librarians on student outcomes is critical as school librarians compete for limited funding and resources in schools (American Association of School Librarians, 2014; Hughes, 2014; Morrison, 2015). Simply put, the sustainability of the school library profession relies upon credible and reliable findings. Although the effect of school librarians on student achievement has been investigated for over two decades, the vast majority of the studies have relied upon descriptive and correlational analyses (Morris & Cahill, 2017). Despite the fact that frequently cited state-commissioned school library studies have not withstood the peer review process, their findings have shaped many practitioners' and researchers' beliefs that "the mere presence of a librarian is associated with better student outcomes" (Johnston & Green, 2018; Lance & Kachel, 2018, p. 17; Lance, Rodney, & Hamilton-Pennell, 2001). Although correlational research may reveal strong associations that may prove useful in the design of interventions to test causation in experimental studies and allow researchers to manipulate the independent variable (Conn, 2017), correlational analyses do not take into account other underlying variables that may be causing or confounding the effect.

In 2014, the American Association of School Librarians (AASL) announced a need for a "credible way" (p.9) to provide evidence of the "positive influence" of state-certified school librarians on student learning (p. 4). To work towards that goal, AASL, with funding provided by the Institute of Museum and Library Services (IMLS), invited 50 scholars across a broad spectrum of disciplines to explore how researchers could design research to support causal inference. A white paper summarizes the outcomes of the 2014 CLASS Summit and acknowledges that although "over 25 correlational studies of library effects on student and teacher outcomes" had been "valuable in identifying possible effects and the features of libraries and librarians that may cause them, [the studies] are generally not able to rule out plausible alternative explanations in a credible way" (American Association of School Librarians, 2014, p. 15). Calls for accountability and shrinking school budgets place pressure on school librarians to provide evidence of what they do, in terms of instructional activities, or what they provide, in terms of informational resources, and how such offerings and/or resources influence student outcomes. In order to establish a credible link between the hypothetical cause (the school librarian) and the effect (improved student achievement), researchers must demonstrate that they have investigated and rejected all other plausible explanations except the investigated causal one (Murnane & Willett, 2011). This has yet to be accomplished in school library research.

Threats to Scientific Integrity: The Legitimacy of Published Results

It is critical for scholars exploring ways to provide credible evidence of the school librarian effect to be aware of discussions across the wider scientific community concerning pervasive errors and biases in data collection, analyses, and publication of research results that threaten the reliability of contemporary scientific inquiry. The widespread prevalence of such weaknesses, empirically confirmed by researchers across disciplines (Button, Bal, Clark, & Shipley, 2016; Higgitt, 2012; Ioannidis, 2005; Jha, 2012; Wicherts, 2017), leads many researchers and statisticians to question the trustworthiness and legitimacy of published results (Ekmekci, 2017; Francis, 2013; Gelman & Loken, 2014; Ioannidis, 2005). As similar discussions concerning these issues have not taken place within the school library research community, in this paper we explore two of these threats within the context of school library research. It is hoped this will foster an open and candid conversation along with critical reflection on commonly held school library beliefs.

The Challenge of School Library Research

For nearly two decades a preponderance of school library studies continue to uphold the belief that the "mere presence of a school librarian" improves student achievement (Lance, 2001; Lance & Kachel, 2018; Lance et al., 2001). However, this simplistic stance ignores the multifaceted role of today's state-certified school librarian and the large number of overlapping, interwoven, and often confounding variables that exist within the learning environment. "The number of different players who contribute to education, and the complexity of their interactions, make it difficult to formulate parsimonious, compelling theories about the consequences of particular educational policies" (Murnane and Willett, 2011, p. 19). Although it is difficult to isolate the influence of the school librarian apart from all other activities and academic interventions, school, student, and library-related variables must be included in investigations in an effort to rule out other plausible alternative explanations and establish a credible link between the hypothetical cause (the school librarian) and the effect (student achievement). Although it is difficult in quasi-experimental and observational research to rule out all plausible competing explanations for the hypothesized link between cause and effect, for "each rival explanation that you do succeed in ruling out explicitly, the stronger is your case for claiming a causal link between treatment and outcome" (Murnane & Willett, 2011, p. 38).

The reality is that school librarians often work in a school building (or multiple buildings), across grade levels and disciplines, and with a range of student populations, teachers, administrators, educational staff, aides, parents, guardians, and community stakeholders. School librarians are employed in a variety of community settings (i.e., rural, suburban, and urban), and may or may not have access to community and cultural resources. School librarians may have access to a wide range of physical and electronic resources, or a limited number. School librarians' schedules may be fixed, flexible, blended, or a combination thereof. Each scheduling configuration, along with the nature of instruction that takes place and the instructional resources utilized, needs to be examined to try and determine what effect the intervention--what the school librarian may do (e.g. the amount of time the school librarian spends with students (the independent variable) has on student learning (the dependent variable). Student demographics, building climate, prior student achievement, and school library variables (certification, full-time status, years of employment, number of years in the building, educational background etc.,) must be considered along with the instructional, technological, and staffing resources of the district, school, and community. Other factors to consider include whether technology is available to students outside of school (personal devices and/or technology provided within the community i.e., school, public libraries, and/or afterschool centers), and the amount of time school librarians spend with students and teachers. Many of these factors may be determined by building or district administrators who often exert influence on the collaborative culture of a school and the level of support that the school librarian receives (Huguet, 2017; Ketterlin-Geller, Baumer, & Lichon, 2015). Furthermore, school librarians engender different dispositions (values, beliefs, attitudes, and commitment), educational backgrounds, professional experiences, instructional styles, technological expertise, and beliefs regarding how students learn. These constructs and combinations of variables have the potential to influence student achievement, making it difficult to understand how the "mere presence" of a school librarian is all that is required to improve student performance. So how and why has the field come to rely on such a simplistic explanation?

Publication Bias

Publication bias--the tendency to publish studies with positive findings with more frequency than those with negative or inconclusive results--is prevalent throughout scientific literature (Bial, 2018). In an oft-cited study, Ioannidis (2005) uses Bayes' theorem and reports that more than half of published research findings are false. While not all researchers agree with the...

To continue reading