Recognising quality in reports of chemistry education research and practice

Changes at CERP

Chemistry Education Research and Practice, CERP, has recently undergone a number of changes of relevance to its contributors and readers. One of these is a change in editorship, and on behalf of the journal I would like to both acknowledge the tremendous amount of work undertaken by my predecessors, Georgios Tsaparlis and Stephen Breuer, and to offer them my personal thanks for their support (and good natured tolerance) during the transition period. Under the stewardship of Stephen and Georgios, CERP has become established as an internationally recognised journal of importance to those working in chemical education whether their primary focus is teaching and learning of chemistry at school, college or university level.

This brings me to another key change for the journal. Since 2005, CERP has been published on the Royal Society of Chemistry (RSC) website, and has been available freely to anyone with access to the web. The RSC is both the professional body for chemists in the UK and Ireland, and a long-standing learned society. Part of its charitable mission is to further chemical education, and it has published CERP without charging readers or authors. This has enabled work published in CERP to be available to teachers in schools without access to subscription-based academic journals, and to scholars in developing countries where library budgets are minimal. The ongoing success of CERP has convinced the RSC to invest in the journal in a number of significant ways. CERP has now been integrated into the RSC's publishing division, alongside some of the most significant and historically important chemistry journals. A managing editor has been appointed (Karen Ogilvie) from the RSC permanent staff (Karen is also the Editor of the RSC's education magazine, Education in Chemistry), and RSC Publishing are now handling all matters concerning production of articles once they have been accepted for inclusion in the journal.

As part of this development, another change is that all submissions are now being administered through an on-line editorial platform, ScholarOne, which facilitates effective administration of the peer-review and editorial processes. This means that authors are able to check that all files have been correctly uploaded before finalising a submission, which is then acknowledged and assigned a manuscript reference almost instantaneously. This system also helps maintain a database of the many referees who contribute to the work of the journal through their careful reading and critique of submissions, and the advice they offer to the editor.

Call for referees

The work of referees is of utmost importance to a research journal, as they are the fundamental resource that makes peer-review possible. CERP is lucky to have been supported by many conscientious and generous colleagues within the chemical education community. The ScholarOne system requires referees to be registered with the system before they can be approached to review a submission (as all the invitations are issued by the system), and allows referees to manage their own record, indicating areas of special interest and availability to review. Registration is also necessary to submit an article for consideration by the journal. I am therefore inviting all those who are active as researchers in chemistry education, to visit the journal's ScholarOne site, at http://mc.manuscriptcentral.com/rp, to register with the journal, and leave details of your areas of particular expertise and interest.

Papers submitted to CERP are initially screened by the editor, who may decline to send manuscripts for review if they are either considered outside the scope of the journal or clearly do not match the expectations for publication in the journal (for example, minimal citation of relevant literature; or papers that are purely descriptive or lack clear empirical content). CERP always uses at least two referees, and the editor then scrutinises the submission informed by referees' comments, before deciding to reject a paper, accept it for publication, or request a revision from the authors. The review process is designed both to ensure the quality of published material, and to support authors in improving work suitable for publication.

Evaluating contributions to the chemistry education literature

As CERP is looking to build up its college of reviewers, it seems appropriate to offer some guidance, especially for new referees, on what a referee should look for in a paper that will be published in an international research journal. Some quality criteria (Eybe and Schmidt, 2001) will apply to any submission, but as CERP invites different types of contributions, this will also determine what a referee should be looking for in a particular submission.

CERP is currently inviting articles that are:

• papers that offer research reports (including reports from chemistry teachers undertaking and researching innovative practice)

• theoretical perspectives on chemistry education

• reviews of chemical education research, or of other areas of research of direct and clear significance to the teaching and learning of chemistry

Common criteria that would apply in each of these cases include quality of argument, novelty, and linkage to previous scholarship.

Quality of argument

A key feature of any publishable contribution is that it should be clearly argued, so that any knowledge claims made can readily be seen to follow from the presentation of empirical data and/or analysis of literature offered. This requires logical argument, clarity of English, and careful organisation of writing. CERP does not require authors to necessarily use standard headings such as ‘introduction’ ‘methodology’, ‘findings’ and ‘discussion’ (although these will often be suitable). What is expected is that writing is arranged so that there are helpful section headings and a clearly signposted flow of argument for readers to follow.

Many empirical studies are organised around one or more precise research questions that set out the particular objectives being addressed in the paper. The research questions can be considered as a kind of focal point or fulcrum in the writing, as they should be shown to be motivated by the literature reviewed in the introductory section of the paper, and the research design should be appropriate to address them. At the end of the paper the author(s) draws conclusions from the study and considers its implications for practice of further research, and the reader. These conclusions should reflect the extent to which progress has been made in the study towards answering the research questions: something a reader can check because the study's findings are discussed in sufficient detail. It is not always necessary for research questions to be fully answered for a paper to be of publishable standard: partial answers, or indeed accounts of unexpected complications impeding anticipated progress, may often make a useful contribution to the community's understanding of a topic. However, it is always important that there is coherence throughout the manuscript, and any knowledge claims are appropriately tempered in relation to the limitations of the methodology, the context of the research, and what can logically supported by the evidence marshalled in the study.

Novelty in work of publishable quality

Novelty means that a paper cannot simply report or discuss knowledge that is already published, but rather needs to go beyond what is already reported in the literature. Now, scientific literature builds up knowledge by a cumulative process, so a referee needs to make a judgement about the degree of novelty of a submission. Novelty could mean taking a perspective that is well developed in other fields, but not yet established in chemical education, and discussing how it might be applied in teaching chemistry or enquiring into teaching and learning in chemistry. A review may show novelty by offering an analysis or synthesis of existing literature that goes beyond simply cataloguing existing publications on a topic, to reveal a previously unnoticed pattern or trend across published studies.

In empirical work, novelty might be quite nuanced. For example it may be that a ‘replication’ study is worth reporting in the literature, if, for instance, it suggests that findings first reported in one educational context may not transfer to a different context elsewhere (different curriculum context, age group, etc.). It may also be worth reporting that findings from one context do seem to be generalisable to other contexts—if those contexts are dissimilar enough for such a finding not to appear obvious. However, referees are unlikely to consider a replication of a published study in a very similar context (that does indeed replicate previous findings) to justify publication without some additional rationale (e.g., perhaps the previous work has been the subject of published critiques questioning its findings; perhaps some time has passed and there have been claims in the interim that previously identified problems have since been addressed). More commonly a paper will not set out to simply replicate prior research, but to move beyond it by following up ideas and questions deriving from previous studies.

Linkage to prior research

The development of any field is iterative, with each new study offering an incremental step that builds upon existing knowledge and understanding in the field. That is, research and scholarship needs to be located within research programmes that have established the shared commitments informing researchers, and which offer guidance on the strategic priorities for work in the field (Taber, 2009). In a well-established discipline such as chemistry, some core commitments are so commonly shared that authors may take them for granted (that is, contemporary research papers in chemistry are not expected to cite primary literature to justify using atomic theory, or core concepts such as elements and compounds). However, reports in particular specialisms are required to cite the specific literature that provides the rationale for the study being reported.

When reading an account of an empirical study, readers will normally expect the literature review to set out a conceptualisation of the issue or problem being studied. This usually motivates the specific research question(s) that are being addressed in the study. As there is often a range of potential perspectives, each of which could offer ‘alternative lenses’ for exploring a particular problem or issue (Alsop et al., 2005), readers expect the authors to set out and explain the particular theoretical perspective that informs the study. This may be more important in chemistry education than in other areas of chemical science, as the complexity of teaching and learning is such that often each of a range of quite distinct perspectives can offer valuable, complementary insights into different facets of the same educational phenomenon.

The literature should be used critically and strategically. Authors are not expected to cite every study they have found on the topic of their paper, but rather to set out the theoretical framework for their study, by citing (a) seminal studies that set up the topic area, and (b) those previous studies that are of particular relevance to the rationale and/or approach adopted for the study being reported. The author needs to establish that the current state of the literature demonstrates that the study reported is potentially able to make a substantive contribution to the literature: that is, that it is relevant to, and able to offer new knowledge claims about, an identified corpus of literature relating to chemistry education.

Appropriate methodology

There is a wide range of methodologies that are admissible in chemical education research (National Research Council Committee on Scientific Principles for Educational Research, 2002), and CERP does not show a bias towards particular approaches. Rather, the authors of each submission need to persuade referees that their research design offers a coherent and suitable approach to tackling the research question(s) motivated by the conceptual framework set out in the review of previous literature, and the theoretical perspective being adopted (Bodner and Orgill, 2007). All methodologies have strengths and limitations, and they differ both in the types of research questions they can be used to address, and the types of knowledge claims they can lead to.

So, for example, experimental design can be very powerful, but relies upon statistical techniques, some of which assume population parameters (that may rarely apply in real classes or cohorts), and which normally require randomisation to treatments (which is not always possible when researching existing classes and programmes). Even when randomisation is possible, there are well know threats to validity that plague experimental approaches in education: students respond to novelty, teacher expectations and confidence, and to the attention of researchers collecting data. Unfortunately double-blind research that could avoid these problems is rather difficult to organise in educational contexts! So studies which compare, for example, two parallel classes taught by traditional and innovative approaches, may offer highly significant results, and even substantial effect sizes, without always convincing the reader that it was the innovative nature of the teaching that was responsible for the positive outcomes. Authors should acknowledge the limitations of their approach, whilst offering a strong case for the claims they make. As educational phenomena are so complex, we often have to accept that empirical work is seldom as definitive in chemistry education as it can often be in chemistry itself.

Experimental design is only relevant to ‘confirmatory’ research where there are specific hypothesis to test, based on instrumentation tied to well specified constructs. At the other end of the scale are exploratory ‘discovery’ studies (Biddle and Anderson, 1986) using open-ended (rather than pre-structured) approaches intended to develop interpretations that help us understand educational processes: for example case studies of particular learners' understanding, or of student dialogue in a lesson, or grounded theory studies looking to explain poorly understood educational phenomena. These types of approaches may offer great insights into the particular cases and contexts they study: but often raise questions about generalisability (Taber, 2000). These more ‘interpretive’ approaches offer somewhat complementary strengths and weaknesses to experimental approaches that look to test well defined educational hypotheses; and so CERP welcomes well-planned and executed empirical studies of both exploratory and confirmatory nature, as both are potentially able to contribute to the field of chemistry education.

All studies in which human subjects are participants, regardless of the approach taken, need to adhere to ethical guidelines. This is something that it is especially important to bear in mind when research is carried out with one's own faculty colleagues or students, when issues of informed consent and confidentiality may become more problematic. In these situations in particular, the methodological imperative to collect full data sets and report detailed information may need to be somewhat moderated by the ethical imperative of respect for participants and their choices.

Ultimately all studies are limited, and it is important for authors to frame their claims accordingly, to show both how they add to existing knowledge, and also how they necessarily include provisos that may indicate the directions for further research. The referee has to both critique the arguments made (are the claims justified in terms of the evidence presented?) and judge whether the study offers new knowledge claims that are both robust and substantive enough to justify publication. Few, if any, studies can be considered definitive, so a judgement must be made about whether a study offers the basis to inform future work—either in the practice of chemistry education, or in terms of suggesting directions for further research.

Reporting research into practice

CERP is a peer reviewed research journal that proclaims itself to be about chemistry education research and practice. This raises the question of what kinds of papers about practice justify publication in an international research journal? The short answer is: those that meet the quality criteria discussed above. Empirical papers need to offer new knowledge that is supported by evidence; clearly and coherently argued; and deriving from studies informed by existing literature. Papers make claims from empirical studies that seek to inform research and/or practice in chemical education elsewhere. A ‘practice’ study (perhaps reporting an innovative teaching activity), just as a ‘research’ study (perhaps reporting aspects of student understanding of some chemical concept), needs to demonstrate that it is based on carefully collected and analysed data that does indeed support the claims made. So if a submission reports innovative practice intended to modify the pattern of student learning, improve student attitudes, or increase participation rates in further chemical education, then it should include a sufficiently robust case for the innovation actually having that effect. That means the collection and analysis of suitable data, following an appropriate methodology (just as discussed above). When (as is often likely to be the case) the study reports from a particular course or institutional context, it should include sufficient details of the context in which the innovative practice occurred to allow readers to make judgments about the relevance of the work to their own different professional contexts (Taber, 2007).

It is often argued that the most appropriate way for practitioners to develop their own practice is to use ‘action research’, an approach which puts a greater focus on achieving improvements, than systematically evaluating them. Action research can be a powerful way of improving teaching and learning in a local context. However, because of the lack of systematic documentation and evaluation inherent in action research, and the focus on cycles of successive modifications to practice (usually before extensive data sets relating to any particular iteration become available), it often only facilitates limited theorising about how and why changes have occurred—and so then has modest potential to inform practice beyond the original context. So whilst action research is a worthwhile activity, it usually does not provide the basis for reports in the research literature. This is not to suggest that papers reporting action research will necessarily be excluded from CERP, as it is possible to undertake action research in ways that meet the expectations for publishable research (although often the priority of action researchers is to improve practice rather than to document their efforts). So, for example, it is possible to supplement action research with an additional layer of documentation, and more extensive data collection and analysis, to build a detailed case study of the action research process to support an argument that an innovation has relevance beyond the research context.

Ultimately then, CERP continues to invite a wide range of submissions, relating to all aspects of chemistry education, on the understanding that whatever the type of submissions, manuscripts will be accepted for publication where our peer reviewers consider that they make a contribution to chemistry education by offering an argument for some new knowledge supported by a careful analysis of evidence: be that a review of existing literature, analysis of systematically collected research data, or a careful evaluation of innovative practice.

Keith S. Taber

Cambridge, December 2011

References

1. Alsop S., Bencze L. and Pedretti E., (2005), Analysing exemplary science teaching: theoretical lenses and a spectrum of possibilities for practice, Buckingham: Open University Press.
2. Biddle B. J. and Anderson D. S., (1986), Theory, methods, knowledge and research on teaching. In M. C. Wittrock (Ed.), Handbook of Research on Teaching (3rd ed., pp. 230–252). New York: Macmillan.
3. Bodner G. M. and Orgill M., (Ed.), (2007), Theoretical Frameworks for Research in Chemistry/Science Education, Upper Saddle River, NJ: Pearson Education.
4. Eybe H. and Schmidt H.-J., (2001), Quality criteria and exemplary papers in chemistry education research, Int. J. Sci. Educ., 23(2), 209–225.
5. National Research Council Committee on Scientific Principles for Educational Research, (2002), Scientific Research in Education, Washington DC: National Academies Press.
6. Taber K. S., (2000), Case studies and generalisability–grounded theory and research in science education, Int. J. Sci. Educ., 22(5), 469–487.
7. Taber K. S., (2007), Classroom-based research and evidence-based practice: a guide for teachers, London: Sage.
8. Taber K. S., (2009), Progressing Science Education: Constructing the scientific research programme into the contingent nature of learning science, Dordrecht: Springer.

---