A rationale for publishing peer-reviewed study protocols in the Nordic Journal of Literacy Research to increase scientific rigour

The present editorial article presents the concept and function of peer-reviewed study protocols as part of the academic process, particularly in the context of furthering transparency and rigour in educational research. The aim is to provide a first rationale for publishing peer-reviewed study protocols in the Nordic Journal of Literacy Research (NJLR) in order to further increase scientific rigour in its field, which is now well established, although it is still young. Our proposal goes beyond the status quo by also encompassing qualitative study protocols. Here we have drawn our inspiration from common practices in medicine and from claims within the philosophy of science to the effect that the fate of both quantitative and qualitative studies stands or falls with their transparency. Our proactive suggestion also reflects a ‘climate change’ with regard to the status of qualitative studies on the international scene.

The publication of peer-reviewed study protocols is now common procedure in some fields of research. This also happens in educational research, but much less consistently. A study protocol can be defined as a highly precise and scholarly description of the design of an upcoming study, including its rationale, aims, research questions and measures. Such a protocol has several functions. First of all, protocols ensure transparency and force researchers to adhere to their initial choice of approach. This is of particular importance in quantitative studies, to prevent cherry-picking. Second, protocols give other researchers information at an early stage about the scope and scientific rigorousness of a study. Such early communication helps to avoid duplication of effort and makes it possible to position new studies relative to the one described. When the protocol is not published, other researchers generally have to rely on hearsay. Third, the publication of the research protocol will facilitate the subsequent publication of results and findings, given that the methodology used has already been peer-reviewed. A related advantage is that, as a protocol allows the methodology to be described in great detail, a brief description and a reference to the published protocol may be sufficient in subsequent reporting from the study. This will obviously free up sorely needed space for other things. Fourth, the publication (and the preceding peer review) of the protocol may yield scholarly feedback on the study design and methodology at an early stage of the research process, at a time when it is still possible to enhance the scientific rigour of the study design. Eysenbach (2004) also emphasizes open access journals in this respect.

At present, only a few of the published study protocols in the field of literacy research come from Nordic researchers. Further, those have been published in international journals – peer-reviewed study protocols are conspicuous by their absence in Nordic journals. Moreover, a closer look at the few protocols published by Nordic researchers reveals that they mostly concern certain methodologies, such as randomised controlled trials (Solheim, Rege & McTigue, 2017, Lundetræ, Solheim, Schwippert & Uppstad, 2017), pre-registered meta-analyses (e.g. Rogde, Hagen, Melby-Lervåg & Lervåg, 2016) and – in one case – a mixed-methods study (Mangen, Hoel, Jernes & Moser, 2019). It is true that certain efforts aimed at attaining some of the advantages listed above can be seen in the Nordic context, for instance the communication of non-peer reviewed project descriptions (e.g. Skaug, 2012). However, such communication tends to take place in ‘grey literature’, i.e., outside the system of scientific evaluation (Auger, 1989). Because of the lack of peer review, it is likely that the functions related to increased scientific rigour will usually not be fulfilled. Also, it should be stressed that the idea of a protocol goes far beyond simply conveying information about the authors’ aims in a project description of the kind typically elaborated for the purpose of obtaining funding (e.g. Skaug, 2012).

To remedy this situation, we suggest the establishment of a format for peer reviewed study protocols in the NJLR. The appearance of the NJLR in 2015 was deemed to reflect the fact that Nordic literacy research had reached a certain level of maturity (Skaftun, Solheim & Uppstad, 2015). While our research field can thus be seen as established, it is important to keep in mind that it is not only young but also small from an international perspective – the Nordic countries had a total population of only 27 million people in 2019. However, it is to be hoped that both youth and small size can be compensated for through scientific rigour, and this proposal represents an attempt by the editorial team to do so. We believe that, by establishing such a format, we will help to introduce the concept of published peer-reviewed study protocols to members of our research community who may not yet be familiar with it. If we simply started publishing such protocols without an introduction, reviewers and readers might well find the manuscripts they were sent or the articles they found in the NJLR to be confusing – wondering ‘Why are there no results?’

The idea of published study protocols originates from – and is standard procedure in – a field which is quite distant from that of literacy research, namely medicine. The main rationale in that field for publishing study protocols is that doing so is an optimal way to ensure that the hypotheses, measures and methodology of large studies are stated clearly at an early stage. Hence the use of study protocols is mostly – and historically – tied to quantitative approaches. However, medical research has broadened its methodological scope over time and qualitative studies have become frequent contributors to evidence-based practices. As a consequence, qualitative-study protocols have also become common in the field of medicine (see, e.g., NHS, 2016). Even so, it is surprisingly difficult to find, in the medical research literature, explicit statements of the rationale for drawing up qualitative protocols. One possible reason for this lack of explicit justification is that publishing the protocol may simply be too self-evident a measure to take in order to support scientific rigour. The logic underpinning the ethical guidelines for clinical trials, according to which ‘every clinical trial must be registered in a publicly accessible database before recruitment of the first subject’ (World Medical Association, Inc., 2008, Item 19, p. 3), may simply have spilled over onto qualitative studies without anyone experiencing the need to state the rationale for publishing the protocol.

However, in a field such as that of literacy research, to many of whose practitioners the protocol format and its purpose will be new, there is an obvious need to give such an explicit rationale for publishing the study protocols of qualitative studies. In this context, it is interesting to note that – in a field which would seem to be a little closer than medicine to literacy research – the American Psychological Association (APA) has recently published standards for qualitative research, both to increase scientific rigour and to acknowledge the importance of qualitative approaches alongside quantitative ones (Levitt et al., 2018). These APA standards came ten years after the first corresponding standards for quantitative research (Applebaum, 2018). Those earlier standards were prompted by ‘a mounting concern with transparency in science’ (p. 3) and it was pointed out that their publication was ‘contemporaneous with the development of standards in other fields’ (p. 3). Standards can in fact be seen as pursuing many of the same objectives as study protocols, albeit at a more overall level. The publication of the qualitative APA standards can be considered to reflect a change in the scientific climate in the sense that the choice between qualitative and quantitative methods is increasingly viewed as a methodological one, to be based on an assessment of appropriateness to the research questions asked. This is by no means a new position, but it seems to have taken on greater weight in recent years. According to Thorleif Lund (2005), qualitative and quantitative studies can be seen as parts of the same paradigm, if both are based on critical realism (Popper) and on ideas of general validity (Shadish, Cook & Campbell, 2002). In line with this, the philosopher Dagfinn Føllesdal (1979) has claimed that the hermeneutic method typical of qualitative studies follows the same principles as the hypothetico-deductive method often used in quantitative studies. Finally, a further similarity between qualitative and quantitative studies is that both, as noted by Maxwell (2009), tend to benefit from having their design finalised before they are carried out.

In our view, the above-mentioned new APA standards for qualitative research (Levitt et al., 2018) set out a good rationale for publishing the protocols of qualitative studies as well: they state that ‘the value of transparency is at the root of the reporting standards across qualitative methods’ (p. 29), emphasising that a protocol provides an unparalleled opportunity for transparency, and they characterise the standards as a ‘representation of process’ (p. 28). In line with this last claim, we want the protocols we envisage to be a representation of the research process. We do not intend for those protocols either to standardise that process or to set any limits beyond which it must not venture, but we do want them to represent the research process.

Both Levitt et al. (2018) and Fujiura (2015) note that the description of the methodological approach taken tends to require more space in reports of qualitative studies than in reports of quantitative ones. As already mentioned, the protocol format could be a way to provide a highly detailed and precise description of qualitative studies, which would facilitate the subsequent publication of articles about study findings. Hence the outcome would be both less effort and greater rigour.

When it comes to the format of the protocols for literacy studies that we have in mind, we believe that the definitions and the detailed description of standards given in Levitt et al. (2018) would serve as an excellent foundation, helping authors and reviewers achieve their common goal of enhancing the scientific rigour of Nordic literacy research. Quantitative protocols similarly tend to follow established reporting standards (Applebaum et al., 2018) as well as the practices of the various journals that publish peer reviewed study protocols (see, e.g., the International Journal of Educational Research). There will undoubtedly be a need for some fine-tuning of the format to suit the specific needs of our research field, based on the experience that we will acquire over time, but right now the most pressing need is to build support for the principle of publishing peer reviewed study protocols.