View PDF VersionPrevious ArticleNext Article
 
DOI: 10.1039/D2RP00184E (Paper) Chem. Educ. Res. Pract., 2023, 24, 263-282

Generalized rubric for level of explanation sophistication for nucleophiles in organic chemistry reaction mechanisms†

Brandon J. Yik , Amber J. Dood , Stephanie J. H. Frost , Daniel Cruz-Ramírez de Arellano , Kimberly B. Fields and Jeffrey R. Raker *
Department of Chemistry, University of South Florida, Tampa, FL 33620, USA. E-mail: jraker@usf.edu

Received 20th June 2022 , Accepted 11th October 2022

First published on 25th October 2022

Abstract

Reaction mechanisms are central to organic chemistry and organic chemistry education. Assessing understanding of reaction mechanisms can be evaluated holistically, wherein the entire mechanism is considered; however, we assert that such an evaluation does not account for how learners variably understand mechanistic components (e.g., nucleophile, electrophile) or steps (e.g., nucleophilic attack, proton transfer). For example, a learner may have proficiency of proton transfer steps without sufficient proficiency of a step where a nucleophile and electrophile interact. Herein, we report the development of a generalized rubric to assess the level of explanation sophistication for nucleophiles in written explanations of organic chemistry reaction mechanisms from postsecondary courses. This rubric operationalizes and applies chemistry education research findings by articulating four hierarchical levels of explanation sophistication: absent, descriptive, foundational, and complex. We provide evidence for the utility of the rubric in an assortment of contexts: (a) stages of an organic chemistry course (i.e., first or second semester), (b) across nucleophile and reaction types, and (c) across prompt variations. We, as well, present a case study detailing how this rubric could be applied in a course to collect assessment data to inform learning and instruction. Our results demonstrate the practical implementation of this rubric to assess understanding of nucleophiles and offer avenues for establishing rubrics for additional mechanistic components, and understanding and evaluating curricula.

Introduction

Reaction mechanisms are ubiquitous with organic chemistry. Understanding of reaction mechanisms, both how to draw and the meaning conveyed, is critical for success as an organic chemist (Bhattacharyya, 2013; Nedungadi and Brown, 2021). However, studies have consistently shown that learners have difficulty using the electron-pushing formalism, the language of organic chemistry (Bhattacharyya and Bodner, 2005; Anderson and Bodner, 2008; Ferguson and Bodner, 2008; Kraft et al., 2010; Grove et al., 2012b; Bhattacharyya, 2014). Learners’ ability in interpreting reaction mechanisms influences their learning (Daniel, 2018). Therefore, we are interested in learners’ mechanistic reasoning, which broadly includes learners’ descriptions of how a reaction occurs through the movement of electrons and changes in bonding (Machamer et al., 2000; Russ et al., 2008; Yan and Talanquer, 2015; Bodé et al., 2019; Dood and Watts, 2022).

Sociologists (e.g., Collins, 2011) suggest that knowing language surrounding a practice (e.g., mechanistic reasoning; see Talanquer, 2018; Dood and Watts, 2022) may better advance associated understanding than simply engaging in the practice (e.g., drawing a reaction mechanism). In other words, knowing how to communicate within the context of a practice is necessary for participation in a practice. We assert that educators must aim to cultivate learners’ ability to put their mechanistic reasoning into words (i.e., lexical ability; Connor et al., 2021) when creating reaction mechanism learning experiences; knowing and applying the mechanistic lexicon and language has the potential to promote better understanding of reaction mechanisms.

As Cooper and others have argued, learners must have opportunities to demonstrate and wrestle with any outcome or objective we seek through our instruction (Cooper, 2015; Becker et al., 2016; Stowe and Cooper, 2017; Bodé et al., 2019; Galloway et al., 2019; Watts et al., 2020, 2022). In other words, if we want learners to ascribe meaning to reaction mechanism drawings, then we must ask learners to communicate that meaning, e.g., through written explanations. Scholarship in this area tends to favor single or few step reactions wherein an overall assessment of understanding, reasoning, etc. can be declared (Cooper et al., 2016; Caspari et al., 2018; Bodé et al., 2019; Crandell et al., 2019, 2020). However, even the simplest of reaction mechanisms to draw (arguably the bimolecular substitution reaction, i.e., one-step, two mechanistic arrows, two starting materials, two products), has multiple components that need to be jointly considered when explaining the reaction mechanism drawing; this is noted by Flynn and Ogilvie (2015) in their implementation of a carbonyl-reaction-first approach to learning organic chemistry. As well, research on learner understanding of nucleophiles and electrophiles (two entities “chunked” as one by organic chemists) has shown that deep understanding of one is not indicative of deep understanding of the other (Anzovino and Bretz, 2015, 2016).

Constructed response items provide an important assessment means for capturing the meaning ascribed to reaction mechanisms. Constructed response items with associated evaluative rubrics (e.g., Becker et al., 2016; Cooper et al., 2016; Caspari et al., 2018; Bodé et al., 2019; Crandell et al., 2019, 2020; Deng and Flynn, 2021; Noyes et al., 2022), and sometimes computer-based scoring models (e.g., Dood et al., 2019, 2020a; Noyes et al., 2020; Yik et al., 2021), are now frequently reported in the STEM education literature. As we have asserted before, a significant limitation of these assessment items is the focus on highly specific assessment items and examples or contexts. For example, even though broad concepts such as nucleophiles are being assessed, evaluation of responses are built around specifics (e.g., understanding of nucleophiles in the electrophilic aromatic substitution reaction of benzene with chlorine and iron(III) chloride). While such scholarship is valuable, it is restrictive in that an educator wishing to use the constructed response items in their courses is limited to the specific assessment item and evaluative rubric, i.e., the scholarship does not account for variations in the prompt type or specific example. We argue that such items and associated rubrics need to be defined by concept and not specifics: for example, learning goals for understanding of a nucleophile–electrophile reaction step and an associated rubric have more utility for educators and ultimately promoting learning.

In this work, we report the development of a rubric that evaluates the level of explanation sophistication for understanding of nucleophiles in organic reaction mechanisms through written responses to open-ended formative assessment items. We start from the chemistry education and broader education research literature to define the levels of explanation sophistication including identifying areas of confusion and opportunities for learning noted in the research literature. This work also builds on prior work evaluating learner understanding of acid–base reactions (Yik et al., 2021). In that study, we noted that the language organic chemists use to describe and explain reaction mechanisms is irrespective of the substrates and reagents in the reaction; therefore, the language used to describe mechanistic components can be conceptualized as non-reaction-type or non-reactant(s) specific. The work reported herein advances our understanding of assessing reaction mechanism learning by: (1) operationalizing levels of explanation sophistication for nucleophiles and (2) reporting a generalized rubric for evaluating a mechanistic component (i.e., nucleophiles) in the context of an entire reaction.

Understanding of organic chemistry reaction mechanisms

Bhattacharyya and Bodner (2005) argued that the ability to use and understand the electron-pushing formalism in reaction mechanisms is vital for comprehending organic chemistry. Educators routinely note that drawing reaction mechanisms is a foundational topic in organic chemistry education (Goodwin, 2003; Ferguson and Bodner, 2008; Goodwin, 2008; Bhattacharyya, 2013; Bhattacharyya and Harris, 2018). Results from a recent survey of organic chemistry educators in the United States reaffirmed that understanding of reaction mechanisms is important for learner success (Nedungadi and Brown, 2021). While many educators believe that when students draw a reaction mechanism, they ascribe some meaning to the structures and arrows; many students lack a deep understanding of reaction mechanisms and simply adorn their pictures with arrows because the students saw no benefit from these representations (Grove et al., 2012b). As such, students may use techniques, such as rote memorization and surface-level studying approaches, to construct reaction mechanisms (Ferguson and Bodner, 2008; Cooper and Stowe, 2018). In other words, reaction mechanism “pictures” have little to no meaning for learners.

Studies report that both undergraduate and graduate students have difficulties using the electron-pushing formalism (Bhattacharyya and Bodner, 2005; Anderson and Bodner, 2008; Ferguson and Bodner, 2008; Kraft et al., 2010; Grove et al., 2012b; Bhattacharyya, 2014). We should note that graduate student-level work in this area is often done with learners pursuing graduate-level studies in organic chemistry; thus, if there was a population of chemists that we would expect to have advanced understanding, it would be those learners. Some have argued that this may stem from students having difficulty in simultaneously ascribing meaning to curved arrows and using chemical language to describe what is happening in reaction mechanisms (Galloway et al., 2017; Bhattacharyya and Harris, 2018). In general, students favor product-oriented learning over the deep engagement of core ideas needed in process-oriented learning that are reaction mechanisms (Anderson and Bodner, 2008; Kraft et al., 2010; Grove et al., 2012a; Graulich, 2015).

Learners struggle with reaction mechanisms as a whole, but also often lack skills necessary to reason about individual steps in a mechanism (Graulich, 2015). Cruz-Ramírez de Arellano and Towns (2014) reported that students were unsuccessful in recognizing components of alkyl halide reactions, and thus failed to understand the holistic nature of these reactions. Watts et al. (2020) analyzed students’ written explanations of an acid-catalyzed amide hydrolysis reaction to identify features of mechanistic reasoning; it was found that students were consistent in using appropriate language to describe mechanistic steps. Organic chemistry instructors support the notion that learners need to understand components of reactions, saying that the identification of functional groups and reagents along with their classification as electron donors or acceptors are necessary for students to become proficient in mechanistic reasoning using the electron-pushing formalism (Bhattacharyya, 2013). This indicates that learners need to build foundational skills in the components of reaction mechanisms to understand the whole of reaction mechanisms more meaningfully.

An overwhelming conclusion from the chemistry education research literature is that students rely on memorized pieces of knowledge and information to solve mechanistic problems with little understanding of the fundamental concepts (e.g., acid–base chemistry, nucleophiles, electrophiles) that underpin reaction mechanisms (Ferguson and Bodner, 2008; Graulich, 2015). However, students are more apt to concentrate on structural features over function and reasoning (Domin et al., 2008; Ferguson and Bodner, 2008; Kraft et al., 2010; McClary and Talanquer, 2011; Cruz-Ramírez de Arellano and Towns, 2014; Anzovino and Bretz, 2015; Galloway et al., 2017; Graulich and Bhattacharyya, 2017; Galloway et al., 2019; Dood et al., 2020a; Lapierre and Flynn, 2020; Petterson et al., 2020; Xue and Stains, 2020). Students report that constructing reaction mechanisms is meaningless to them and that instruction focused on how reaction mechanisms proceed with little emphasis on the why (Ferguson and Bodner, 2008). However, expert-level mechanistic reasoning requires integrating multiple electrostatic concepts such as formal and partial charges, polarity, electronegativity, electron density, and also identifying and classifying nucleophiles and electrophiles (Bhattacharyya, 2013).

Understanding of nucleophiles

In this paper, we focus on the understanding of nucleophiles. The research literature supports the notion that nucleophiles and electrophiles are not equally understood (Strickland et al., 2010; Anzovino and Bretz, 2015, 2016; Putica and Trivic, 2016) with learners being more successful in identifying and explaining nucleophiles. While organic chemists may view these two entities as a combination, we note that learners need a more foundational understanding of each as they develop understanding of the relationship between the two. We chose nucleophiles to begin our broader work in this area all-the-while noting that this is merely a first step.

Learners hold alternative conceptions regarding nucleophiles and struggle to articulate conceptualizations about nucleophiles. It is documented that students believe nucleophiles are positively charged, electron deficient, and tend to accept electrons; these are all the opposite of what nucleophiles are and how nucleophiles participate in bonding. In other words, students confuse the properties of nucleophiles with those of electrophiles (Akkuzu and Uyulgan, 2016). In a study by Putica and Trivic (2016), a majority of students made incorrect attempts at defining nucleophiles, again conflating nucleophilic properties with those of electrophiles. Additionally, chemistry undergraduates at the end of their four-year degree studies have been reported to struggle in articulating their conceptions of implicit chemical properties, such as nucleophilicity and the electronic interactions that drive chemical reactivity (DeFever et al., 2015).

Students use rote memorization and surface-level practices to identify nucleophiles and nucleophilic behavior (Anzovino and Bretz, 2015). Students also engage in rote memorization of surface-level features related to nucleophilic behavior rather than engaging in deeper-level relationships between structure and reactivity (Anzovino and Bretz, 2015). In another study, students were more successful at identifying nucleophiles than electrophiles (Strickland et al., 2010) and are prone to prioritize structural features to identify nucleophiles in reaction mechanisms (Anzovino and Bretz, 2015, 2016; Weinrich and Sevian, 2017). However, students generally struggle at identifying nucleophiles without a reaction mechanism; this suggests that students are not using mechanistic reasoning when engaging with reaction mechanisms (Strickland et al., 2010).

Students have difficulty relating nucleophiles to other chemical concepts. The development of an assessment of concepts important for developing proficiency in organic reaction mechanisms showed that items related to nucleophiles are difficult; this is compounded with items related to resonance, inductive effects, and intermediate stability being even more challenging (Nedungadi et al., 2020). Multiple studies have reported that students have difficulty in being able to distinguish between nucleophiles and bases, and the connection between the two entities (Cartrette and Mayo, 2011; Cruz-Ramírez de Arellano and Towns, 2014; Anzovino and Bretz, 2015, 2016).

Moving toward meaningful assessments

Much of the research literature focused on student understanding of reaction mechanisms has been done by evaluating students’ construction of reaction mechanism drawings and predicting products using the formalism (Bhattacharyya and Bodner, 2005; Kraft et al., 2010; Strickland et al., 2010; Grove et al., 2012b; Flynn and Featherstone, 2017; Galloway et al., 2017; DeCocq and Bhattacharyya, 2019). DeCocq and Bhattacharyya (2019) reported that exercises, such as drawing mechanistic reaction arrows and predicting reaction products, are not successful at eliciting and evaluating reasoning. Numerous studies have concluded that increased cognitive demand from proposing mechanisms may cause students to not use mechanisms when predicting products, and thus further resorting to rote memorization and surface-level approaches to learning (Ferguson and Bodner, 2008; Grove et al., 2012a; Flynn and Featherstone, 2017; Galloway et al., 2017; Cooper and Stowe, 2018). Before expecting learners to ascribe meaning to reaction mechanisms they have drawn, we assert that learners need opportunities to construct meaning from “correct” reaction mechanisms given to them by cultivating their mechanistic reasoning through written formative assessments by scaffolding learning.

Work by multiple research teams (e.g., Cooper, 2015; Cooper et al., 2016; Dood et al., 2018, 2020a; Bodé et al., 2019; Yik et al., 2021; Raker et al., 2023), have asked students what is happening in reaction mechanism and why it happens. This “what and why?” scaffold has been used to prompt mechanistic understanding in acid–base chemistry (Cooper et al., 2016; Dood et al., 2018, 2019; Crandell et al., 2019; Yik et al., 2021), unimolecular nucleophilic substitution reactions (Dood et al., 2020a), and bimolecular nucleophilic substitution reactions (Crandell et al., 2020). Requiring students to explicitly address each step of a reaction mechanism using underlying fundamental concepts will aid students in appreciating the utility of the electron-pushing formalism (Bhattacharyya and Bodner, 2005). The electron-pushing formalism and reaction mechanisms are models and representations that are essential to reasoning in organic chemistry and provide information about the conceptions of chemical transformations, and therefore are used in the construction of explanations (Goodwin, 2003).

Constructing explanations of reaction mechanisms engages students in key scientific practices. The Framework for K-12 Science Education and the Next Generation Student Standards highlight using models, constructing explanations, and communicating information as essential practices for learning and doing science (National Research Council, 2012; NGSS Lead States, 2013). Engagement in these scientific practices, such as constructing explanations, will aid students in deepening their understanding of science. For example, the writing process aids in deeper thinking and engagement with course content (Rivard, 1994; Bangert-Drowns et al., 2004; Reynolds et al., 2012).

Writing assessments using purposeful constructed-response prompts can promote understanding of chemical concepts (Birenbaum and Tatsuoka, 1987; Cooper, 2015; Stowe and Cooper, 2017; Underwood et al., 2018). Open-ended written assessments should prompt students what is happening in the reaction mechanism and why it happens to promote deeper learning and allow instructors to gain deeper insight into students’ understanding (Bell and Cowie, 2001; Cooper, 2015; Cooper et al., 2016; Underwood et al., 2018). While there are many benefits to open-ended written assessments, providing consistent and prompt feedback using such assessments is difficult.

Rubrics are a possible solution (Wolf and Stevens, 2007; Brookhart, 2018). According to Andrade (2000), rubrics are beneficial because they: (i) are easy to use and explain, (ii) convey clear expectations, (iii) provide students with more feedback than traditional assessments, and (iv) support learning through the development of skills and understanding. Rubrics help articulate instructors’ expectations by describing the qualities of an assessment across a continuum and for providing feedback on student work (Andrade, 2000; Brookhart and Chen, 2015). In developing a rubric, Dawson (2017) recommends a broad set of design elements that should be considered. This objective to assess students can be practically used in, but not limited to, competency-based grading (see Voorhees, 2001) and specifications grading approaches (see Nilson, 2015). In these grading systems, the instructor defines a passing or proficiency threshold on an assessment, for example, reaching a certain level in a rubric for understanding of nucleophiles in a reaction mechanism, and the token system in specifications grading provides students with options to improve and resubmit work that does not meet satisfactory criteria (see Nilson, 2015; Howitz et al., 2021). Specifications grading has been used in various disciplines, including lecture-based chemistry courses (e.g., Diegelman-Parente, 2011; Boesdorfer et al., 2018; Martin, 2019), and therefore, rubrics provide a means to assess such learning in a variety of classroom contexts.

Rubric for evaluating understanding of nucleophiles

Students’ explanations of reaction mechanisms have been evaluated using a variety of frameworks: levels of complexity of relations (Caspari et al., 2018), Chemical Thinking Learning Progression (Bodé et al., 2019), causal versus mechanistic (Cooper et al., 2016; Crandell et al., 2019, 2020), and levels of explanation sophistication (Dood et al., 2020a). For a more comprehensive review of reasoning frameworks, see a review by Dood and Watts (2022) on mechanistic reasoning in organic chemistry.
Levels of complexity of relations. Caspari et al. (2018) characterized students’ comparisons of the activation energy of two mechanistic steps rooted in the complexity of relations (i.e., low, middle, and high). Relations with low complexity use explicit structural differences as a cause to describe relation, or non-electronic effects, to explain change. Relations with middle complexity use explicit structural differences to infer implicit structural causes that are used to describe relation, or non-electronic effects, to explain change. Relations with high complexity use explicit structural differences to infer implicit structural causes that are used to describe an electronic effect to explain change.
Chemical thinking learning progression. Bodé et al. (2019) explored students’ explanations of the likelihood of reaction progression between two reaction mechanisms. The Chemical Thinking Learning Progression framework (Sevian and Talanquer, 2014) was used to characterize these explanations. The framework characterizes responses by mode of reasoning: descriptive, relational, linear causal, and multicomponent causal. Descriptive responses state explicit features without further explanation. Relational responses use explicit and implicit features to draw connections, but do not explain a cause–effect relationship. Linear causal responses build upon relational responses and describe a cause-and-effect relationship using a single variable, and multicomponent causal responses describe cause-and-effect relationships using multiple variables.
Causal mechanistic reasoning. Cooper with others (Cooper et al., 2016; Crandell et al., 2019, 2020) have characterized student reasoning about acid–base and bimolecular nucleophilic substitution (SN2) reactions using a causal mechanistic reasoning framework. Descriptive general responses provide a simplistic description of bond breaking/forming processes whereas descriptive causal response also discusses electrostatic interaction and demonstrates understanding of electrostatic attraction. Descriptive mechanistic responses recognize electrons and explains the movement of electrons, whereas causal mechanistic responses build upon descriptive mechanistic by understanding the electrostatic attraction between chemical species.
Levels of explanation sophistication. Dood et al. (2020a, 2020b) reported an emergent framework for evaluating explanations of a unimolecular nucleophilic substitution (SN1) reaction. Responses were characterized into three levels: Level 1 (descriptive), Level 2 (surface-level why), and Level 3 (deeper-level why). Level 1 responses describe reactions using explicit features with no explanation. Level 2 responses use surface level features to explain why the reaction is happening or uses implicit features with no elaboration. Level 3 responses describe why the reaction happens using implicit features that have been inferred from explicit features or uses electronic effects to explain the reaction.

The levels of explanation sophistication rubric proposed herein synthesizes and operationalizes these four frameworks as informed by the research literature on understanding of reaction mechanisms and nucleophiles. Our rubric for evaluating the understanding of nucleophiles is based on a conceptualization of a generalized rubric for levels of explanation sophistication for the understanding of electrophiles reported by Raker et al. (2023). This rubric is general as it can be applied to a variety of written formative assessment items (see specificity in Dawson, 2017). In operationalizing our nucleophile rubric, four hierarchical levels of sophistication were identified: Absent, Descriptive, Foundational, and Complex (Table 1). This rubric builds upon our previous levels of explanation sophistication framework (Dood et al., 2020a, 2020b) with the addition of the Absent level and mirrors other studies in the literature (Sevian and Talanquer, 2014; Cooper et al., 2016; Weinrich and Talanquer, 2016; Caspari et al., 2018; Bodé et al., 2019; Graulich et al., 2019; Crandell et al., 2020; Dood et al., 2020a; Watts et al., 2020) that suggest students’ reasoning abilities progress through a hierarchical relationship. Additionally, we have changed the level names from our previous iteration of the framework so that the level names may be better understood and applied by learners and instructors in regard to description and hierarchy of the levels. This analytic scoring strategy applies a single level for the evaluative criterion of nucleophiles (see evaluative criteria and scoring strategy in Dawson, 2017). We envision that this rubric can be a paper-based copy, a digital-based document file, or be translated into a rubric in an online learning management system (see presentation in Dawson, 2017).

Table 1 Rubric for levels of explanation sophistication for nucleophiles
Level Description of level Key features of level
Absent • No response • The nucleophile is not identified
• Non-normative • The nucleophile–electrophile reaction step is missing from the explanation
Descriptive • Describes the nucleophile engaging in bond forming processes • The nucleophilic molecule or atom is identified
• Simplistic description of bond forming processes • Electrons are not used to describe nucleophilic behavior
• Nucleophilic behavior is described at a surface, atomic level • Bond forming processes are described (e.g., “nucleophile attacks/forms a bond with electrophile”)
Foundational • Nucleophilic behavior is described at a surface, electronic level • Electrons are central to nucleophilic behavior and arrows represent the movement of electrons
• Explicit features are mentioned • Example descriptors of electrons: Sigma electrons/bond, pi electrons/bond, lone pair (and not alkene or double bond)
• Implicit features may be mentioned, but not fully explained • Bond forming processes are described using electrons (e.g., “the lone pair on the nucleophile attacks/forms a bond with electrophile”)
Complex • Describes why the nucleophile is involved in bond forming processes • Implicit electronic features are used to describe nucleophilic behavior
• Nucleophilic behavior is described at a deeper, electronic level • Examples of electronic properties: electron density, electronegativity, partial charges, molecular orbital descriptions (e.g., HOMO/LUMO)
• Explicit features are used to infer implicit features that are sufficiently explained • Bond forming processes are described using electrons and electronic properties (e.g., “the high electron density on the nucleophile is attracted to the area of low electron density on the electrophile”)

The Absent level has been left out of prior frameworks for evaluating understanding of reaction mechanisms, by us and others (Cooper et al., 2016; Bodé et al., 2019; Crandell et al., 2019, 2020; Dood et al., 2020a, 2020b). It is included here as we note that learners can and do respond with “I’m not sure” or “I haven’t studied this yet” when asked to explain (Dood et al., 2020a). Thus, an assessment tool for practical use in instruction needs a means to characterize and address such responses. The Absent level also addresses instances where explanations of reaction mechanisms may glance over a mechanistic step or provide no understanding about a concept: For example, “the product is formed through an SN2 reaction;” this response does not address the nucleophile in the reaction, nor understanding of nucleophiles in a bimolecular substitution reaction.

The Descriptive level characterizes simple narratives of bond forming processes or states what is happening without any explanation of why: “the ethoxide attacks the carbon with the bromide” and lacks recognition of electrons. Here, students are associating the general language used to describe reaction mechanisms using chemical names and processes with a primary focus on explicit structural features (see descriptive in Bodé et al., 2019; see descriptive general in Crandell et al., 2020; see Level 1 in Dood et al., 2020a, 2020b). A key component at this level is that responses properly identify the nucleophile (Anzovino and Bretz, 2015).

The Foundational level is given to responses that recognize that electrons are responsible for interactions between chemical species leading to bond forming/breaking processes (cf. descriptive mechanistic in Crandell et al., 2020). A response stating, “the lone pair on the ethoxide attacks the carbon with the bromide” is an example at the Foundational level where the key difference is in the addition of the phrase “lone pair.” Implicit features may be mentioned but not fully explained at this level (cf. Level 2 in Dood et al., 2020a, 2020b).

Lastly, the Complex level characterizes responses at a deeper, electronic understanding. These responses describe both explicit and implicit features and includes descriptions of electronic properties or interactions that lead to reactivity, such as electron density, electronegativity, partial charges, and molecular orbital theory descriptions (see causal mechanistic in Crandell et al., 2020; see Level 3 in Dood et al., 2020a, 2020b).

Research goal

Our work is guided by one primary goal: to develop a generalized rubric to evaluate the level of explanation sophistication for nucleophile understanding in organic chemistry reaction mechanisms. We seek to demonstrate that the rubric has utility in a variety of courses (e.g., first or second term), for a variety of nucleophile-containing reactions (e.g., unimolecular nucleophilic substitution reactions or acetal formation reactions), and with a variety of cued assessment prompts.

Methods

This work was conducted under application Pro#00028802, “Comprehensive evaluation of the University of South Florida's undergraduate and graduate chemistry curricula,” as reviewed by the University of South Florida's Institutional Review Board on December 13, 2016; the activities were determined to not constitute research involving human subjects per Institutional Review Board criteria and thus approval was not needed.

Constructed response items

The constructed response item format used in this study was initially reported for use in a single-step proton transfer reaction (Cooper et al., 2016), and modified and expanded for use in other acid–base (Dood et al., 2018, 2019; Crandell et al., 2019; Yik et al., 2021) and multi-step nucleophilic substitution reactions (Crandell et al., 2020; Dood et al., 2020a, 2020b). This format includes asking respondents to describe what is happening in a reaction and to explain why the reaction occurs. In total, 85 reaction mechanisms were used in this work. A summary of these reaction mechanisms is given in Table 2; example reaction mechanisms are provided in Fig. 1 (see exemplars in Dawson, 2017). A full list of assessment items is located in the ESI.†
Table 2 Reaction mechanisms used
Functional group Reaction Variations Number of explanations
Alkyl halide Bimolecular nucleophilic substitution (SN2) 4 820
Unimolecular nucleophilic substitution (SN1) 8 5050
Alkene Halogenation/hydrohalogenation 5 1262
Halohydrin formation 3 717
Hydration/dehydration 4 1276
Alkyne Alkylation 1 411
Alcohol Conversion to good leaving group, then SN2 4 932
Epoxidation/ring-opening 6 1284
Aromatic ring Acylation 1 197
Addition–elimination 1 136
Alkylation 1 196
Azo coupling 1 170
Electrophilic aromatic substitution 6 657
Nucleophilic aromatic substitution 2 225
Carbonyl Acyl substitution 3 403
Aldol addition/condensation 2 273
Alpha-halogenation 1 68
Condensation of an ester/diester 2 402
Conjugate addition 2 255
Enamine/imine synthesis 2 367
Esterification 1 202
Grignard 6 961
Hemiacetal/acetal formation 4 885
Hydration/dehydration 2 137
Hydrolysis 1 70
Reduction 9 1825
Saponification 1 202
Conjugated diene Electrophilic addition 2 553


image file: d2rp00184e-f1.tif
Fig. 1 Example reaction mechanisms.

Two prompt variations were used to collect explanations (Table 3). These are labeled Original and More Cued to convey the intent of changes employed in the second prompt. In the Results and Discussion, we will discuss the intent of each prompt and any observable differences in response patterns.

Table 3 Constructed response item prompt variations
Original More cued
Part A: Describe in full what you think is happening on the molecular level for this reaction. Be sure to discuss the role of the reactant and intermediate. Part A: Describe in full detail the sequence of events that occur at the molecular level for this reaction. Be sure to discuss the role of each reactant and intermediate.
Part B: Using a molecular level explanation, explain why this reaction occurs. Be sure to discuss why the reactants form the products shown. Part B: Using a molecular level explanation, explain why each of the reactants and intermediates interact.

Data collection

Data were collected from seven semesters (Fall 2017, Spring 2018, Fall 2019, Spring 2019, Fall 2020, Spring 2020, and Fall 2021) of the first semester (Organic Chemistry 1) and two semesters (Spring 2020 and Fall 2021) of the second semester (Organic Chemistry 2) of a year-long organic chemistry course at the University of South Florida, a large, research-intensive, public university in the southeastern United States. These courses were taught by four instructors: authors DCR, KBF, and JRR, and F. Costanza (see Acknowledgements). Data were collected in author DCR's Organic Chemistry 1 course in Spring 2018, Fall 2019, Spring 2019, Fall 2020, Spring 2020, and Fall 2021. Data were collected in author KBF's Organic Chemistry 1 course in Fall 2017, Fall 2019, Fall 2020, and Fall 2021; and Organic Chemistry 2 course in Spring 2021. Data were collected in F. Costanza's Organic Chemistry 2 course in Spring 2020. Data were collected in JRR's Organic Chemistry 1 course in Fall 2017, Fall 2019, and Spring 2020; and Organic Chemistry 2 course in Fall 2021. For both courses (Organic Chemistry 1 and Organic Chemistry 2), between Fall 2017 and Spring 2019, the textbook used was Solomons et al.'s Organic Chemistry, 12th edn(2016); between Fall 2019 and Spring 2021, the textbook used was Klein's Organic Chemistry, 3rd edn(2017), and Klein's Organic Chemistry, 4th edn(2021) was used in Fall 2021. For all semesters, constructed response items used the Original prompt; data collected in Fall 2021 used the Original and More Cued prompts.

Constructed response items were administered via Qualtrics surveys. In total, 85 unique reaction mechanism prompts were administered (see Table 2) resulting in 19[thin space (1/6-em)]936 responses. Participants received extra credit towards their term or final examination grade for completing the assessment. Study participants may have completed up to four surveys in a given semester and participants may have completed assessment items in both their Organic Chemistry 1 and Organic Chemistry 2 courses. Some data for unimolecular nucleophilic substitution reactions have been analyzed analogously and reported in other studies (Dood et al., 2020a, 2020b).

Results and discussion

The value of a rubric is in its use as an assessment tool to provide feedback to students and evaluative information for instructors. In addition, students are able to use a rubric to monitor their learning and instructors are able to use a rubric for grading or evaluation (Panadero and Jonsson, 2013). Here we describe the use of our nucleophile rubric in the context of postsecondary Organic Chemistry 1 and Organic Chemistry 2 courses.

Refining and operationalizing the rubric levels

Our levels of explanation sophistication rubric is reported in Table 1, above. In applying this rubric, responses from first and second semester courses that explain reaction mechanisms with multiple nucleophile types (e.g., lone pair, sigma, pi), and span a variety of reaction families (e.g., aromatic reactions, nucleophilic additions, reduction reactions) were first analyzed by author BJY. In this process, author BJY applied the rubric (Table 1) and grouped similar responses based on the language used to explain nucleophiles; characteristics of these responses that distinguished responses from others were noted. Through peer debriefing discussions (see Lincoln and Guba, 1985), author BJY presented descriptions, key features, and examples of responses in each of the four levels of the rubric to authors SJHF and JRR in which these aspects were refined, and inclusion criteria were solidified; this led the characterization of key features for each level in the nucleophile rubric (see “key features of level” in Table 1).

To ensure reliability of the data obtained by using the rubric, interrater reliability checks were performed (see the quality processes criteria in Dawson, 2017). Two raters were used in the interrater process; authors BJY and SJHF are chemistry graduate students in chemistry education research and have served as teaching assistants for at least one of the organic chemistry courses. As the rubric should be accessible and useful for a variety of learners and educators, authors BJY and SJHF are well suited for establishing judgment complexity criteria (Dawson, 2017). Author BJY applied the rubric (Table 1) using the refined key features (Table 1; see Appendix 1 for complete rubric including exemplars) to classify all 19[thin space (1/6-em)]936 responses. For reactions that have two nucleophile interaction components, the higher of the two individual levels was used to classify a response. Author SJHF then independently classified a set of 200 randomly selected responses (∼1% overall) representative of the distribution of nucleophile type and reaction family type of the entire data set. Authors BJY and SJHF initially agreed on 53% (n = 107) of the responses with a weighted (equal weights) kappa of 0.37; after discussion about disagreements, author BJY changed classifications for 10% (n = 20) of the responses, and author SJHF changed classifications for 35% (n = 71) of the responses. There was a final agreement of 96% (n = 193) with a weighted (equal weights) kappa of 0.94. As a result of these discussions, author BJY reevaluated responses with a particular focus on responses classified under the Complex level for unimolecular substitution reactions and reactions involving a carbocation intermediate, and the rubric was further refined to better convey inclusion criteria for different levels which resulted in the rubric and key features as presented in Table 1 (see Appendix 1).

To demonstrate reliability of the data using the refined rubric (see Table 1), a second round of interrater reliability evaluation was performed. Two raters were used in the interrater process: author BJY and an organic chemistry instructor. The refined rubric (Table 1) was applied to a set of 200 randomly selected responses (∼1% overall), different from the data set used in the original interrater reliability evaluation. The two raters agreed on 87% (n = 174) of the responses with a weighted (equal weights) kappa of 0.81; disagreements differed no more than one level.

Rubric use by organic chemistry 1 and organic chemistry 2 courses

The distribution of revised and finalized classifications for the level of explanation sophistication for nucleophiles for the overall data set from Organic Chemistry 1 and Organic Chemistry 2 contexts are provided in Table 4. The distribution for these data sets show that most responses are centered within the middle of the rubric (i.e., the Descriptive and Foundational levels) with fewer students at the extreme ends of the rubric (i.e., the Absent and Complex levels). When describing this data by course (i.e., Organic Chemistry 1 and Organic Chemistry 2), this same distribution pattern holds.
Table 4 Distribution of levels of explanation sophistication for nucleophiles
Level Organic chemistry 1 Organic chemistry 2 Overall
n 13[thin space (1/6-em)]559 6377 19[thin space (1/6-em)]936
Absent 1788 (13.2) 1019 (16.0) 2807 (14.1)
Descriptive 7407 (54.6) 3473 (54.4) 10[thin space (1/6-em)]880 (54.6)
Foundational 3842 (28.3) 1605 (25.2) 5447 (27.3)
Complex 522 (3.9) 280 (4.4) 802 (4.0)

Scale attenuation effects occur when a response format includes too few options to reflect, in this case, response sophistication about nucleophiles. Two types of scale attenuation effects are floor and ceiling effects. Floor effects describe when too many responses are at the lower limit of the scale and ceiling effects describe when too many responses are at the upper limit of the scale (see Šimkovic and Träuble, 2019); scale attenuation effects are considered problematic in measurement theory because the measured variable becomes insensitive to changes in the latent variable and affects the performance of statistical models. A distribution of 15% at the floor or ceiling of a scale has been historically accepted as demonstrating moderate scale attenuation effects (McHorney and Tarlov, 1995); however, this standard has been widely used in the medical community where floor or ceiling effects can have substantive effects on patient health. While our results show nearing a floor effect, our scale is sufficient as the Absent level describes the lack of the concept of nucleophiles in a response and the next level in the scale (i.e., Foundational) describes the next logical step where the response acknowledges the existence of a nucleophile. Additionally, other frameworks describe the highest level as containing multicomponent, causal, or electrostatic descriptions, which are similar to our Complex level, and are based in three to six level scales (Caspari et al., 2018; Bodé et al., 2019; Crandell et al., 2020). Moreover, these frameworks suggest that these ideas are the highest observable descriptions at the target course level. Therefore, we argue that a ceiling effect would suggest that student responses are at the highest level of sophistication and meet or exceed instructor expectation.

Table 5 shows exemplar response excerpts for the levels in the rubric for the concept of nucleophiles (see exemplars in Dawson, 2017). Reactions A through E given in Fig. 1 correspond respectively to those reactions noted in Table 5. Here, we provide exemplars of all levels for a single reaction (Example 1; responses from Organic Chemistry 1) and exemplars for different reactions at each of the levels (Example 2; responses from Organic Chemistry 2).

Table 5 Example excerpts from responses for the levels of explanation sophistication for nucleophilesa
Level Example 1 Example 2
a Reactions A through E correspond respectively to reactions A through E in Fig. 1.
Absent [Reaction A] “The first step of the SN2 reaction is a nucleophilic attack followed by the loss of a leaving group. The interaction happens at the same time.” [Reaction B] “The phenyl magnesium bromide is the Grignard reagent used in this Grignard reaction. The Grignard reagent turns the benzoic acid into a carboxylic acid.”
Descriptive [Reaction A] “The iodine is acting as a leaving group and detaching from the 1-iodopropane. At the same time, the cyanide is attacking the molecule and substituting the leaving group. The reactants and intermediates interact because this is a substitution reaction. Iodine is a good leaving group so therefore this triggers the substitution reaction. Prompting the cyanide to replace it.” [Reaction C] “The negatively charged OCH3 will attack the carbonyl group, pushing one of the bonds up onto the O making it negatively charged. A lone pair will come back down to make a double bond again and the Cl will leave negatively charged.”
Foundational [Reaction A] “The I is a good leaving group and it is leaving while the electron pair on the carbon connected to the triple bond is attacking the carbon that the leaving group is leaving from. They all interact in the same step simultaneously.” [Reaction D] “The alcohol group removes a hydrogen which creates a double bond and makes the carbonyl oxygen single bonded with a negative charge. The electrons in the double bond attack the other carbonyl carbon forming a bond between the two molecules. The electrons on the negatively charged oxygen attack a hydrogen forming it into an alcohol.”
Complex [Reaction A] “The cyanide is acting as a nucleophile attacking the partially positive alpha carbon kicking off the iodine leaving group, thus forming the product. The electron rich cyanide is attracted to the alpha carbon because of its partial positive charge.” [Reaction E] “The oxygen molecules on the sulfur molecule are more electronegative therefore pulling out density and making the sulfur atom have a strong partial positive charge. The electrons [in the] benzene ring are very attracted to this charge and attach the sulfur breaking on [one] of the double bonds to oxygen making it a single. This reaction occurs because benzene is a very stable and strong nucleophile and is attracted to strong electrophiles.”

Rubric use by nucleophile type

To provide further evidence for the generalizability of our nucleophile rubric, Table 6 shows the distribution by nucleophile type. Again, most responses reside within the Descriptive and Foundational levels with very few responses at the Complex level. In addition to prompts that feature a single nucleophile type, we also collected responses for prompts that featured combinations of nucleophile types (Table 6); similar distributions are observed which speaks to the further generalizability of the rubric.
Table 6 Distribution of levels of explanation sophistication by nucleophile type
Level Lone pair Sigma Pi Lone pair & sigma Lone pair & pi Sigma & pi
n 12[thin space (1/6-em)]990 2786 2117 532 1426 85
Absent 1623 (12.5) 586 (21.0) 409 (19.3) 50 (9.4) 134 (9.4) 5 (5.9)
Descriptive 7163 (55.2) 1798 (64.5) 913 (43.1) 327 (61.5) 629 (44.1) 50 (58.8)
Foundational 3590 (27.6) 347 (12.5) 718 (33.9) 148 (27.8) 623 (43.7) 21 (24.7)
Complex 614 (4.7) 55 (2.0) 77 (3.7) 7 (1.3) 40 (2.8) 9 (10.6)

Rubric use by reaction family

Responses were also analyzed by reaction family (Table 7). We classified reactions into five major family types: reactions involving a carbocation intermediate, aromatic reactions, nucleophilic addition, reductions reactions, and bimolecular nucleophilic substitution reactions (SN2). Although a type of nucleophilic addition reaction, we segregated reduction reactions as its own family type due to shared commonality in reducing a carbonyl; as all Grignard reactions included reaction with a carbonyl, Grignard reactions are also classified in this reduction reaction category. Similar to our other analyses, we observe the majority of responses are in the middle levels of the rubric, and that there are more responses at the lower extreme than the upper extreme showing that there is room to demonstrate improvement. This pattern is also observed for responses that wrote about a reaction that contained two family types (e.g., nucleophilic addition and SN2).
Table 7 Distribution of levels of explanation sophistication by reaction family
Level Carbocation Aromatic Nucleophilic addition Reduction SN2 Nucleophilic addition & SN2
n 7432 1581 3264 2786 4612 261
Absent 1008 (13.6) 252 (15.9) 493 (15.1) 586 (21.0) 437 (9.5) 31 (11.9)
Descriptive 3757 (50.5) 780 (49.3) 1891 (57.9) 1798 (64.5) 2504 (54.3) 150 (57.4)
Foundational 2238 (30.1) 463 (29.3) 780 (23.9) 347 (12.5) 1544 (33.5) 75 (28.7)
Complex 429 (5.8) 86 (5.5) 100 (3.1) 55 (2.0) 127 (2.7) 5 (2.0)

In prior work with acid–base reactions (Dood et al., 2018; Yik et al., 2021), we observed that students may use analogous language for nucleophiles to describe Lewis bases (Anzovino and Bretz, 2015, 2016). For example, a response could include, “a lone pair on the alkoxide will attack the proton on water.” While this statement describes a proton transfer reaction, the statement relies on language indicative of nucleophiles. In comparing prompts that do contain a proton transfer step with those that do not (Table 8), we observe similar distributions between the two that resemble similar patterns in the other comparisons. This evidence suggests that other aspects of a reaction mechanism do not readily influence how learners describe nucleophiles.

Table 8 Distribution of levels of explanation sophistication by reactions involving proton transfers
Level No proton transfer Proton transfer
n 3160 16[thin space (1/6-em)]776
Absent 357 (11.3) 2450 (14.6)
Descriptive 1653 (52.3) 9227 (55.0)
Foundational 996 (31.5) 4451 (26.5)
Complex 154 (4.9) 648 (3.9)

Rubric use by prompt variations

The last validity evidence for our rubric is the level of explanation sophistication for nucleophiles collected with multiple prompt types. In the midst of our data collection reported herein, Crandell et al. (2020) modified their original prompt to investigate how wording might affect the responses by activating different resources. This is corroborated by a study that found that students only considered nucleophiles and electrophiles when prompted to do so (Cartrette and Mayo, 2011). In the context of work reported herein and in response to Crandell and colleagues’ work, we changed “explain why this reaction occurs. Be sure to discuss why the reactants form the products shown” in the Original prompt to “explain why each of the reactants and intermediates interact” in the More Cued prompt to make explicit that we would like for responses to explain why for each mechanistic step.

Responses in our work were collected primarily with the Original prompt (see Table 3), with some collected using the More Cued prompt. The distribution of responses among the levels (Table 9) is again similar between the two prompt types and parallels our prior analyses. It is incumbent upon us to note that the data, as reported in Table 9, were not collected per an experimental design that would allow for comparison of response distributions between the two. We recognize that such work would be important; however, our data set does not allow for such an analysis.

Table 9 Distribution of levels of explanation sophistication by prompt type
Level Original More cued
n 15[thin space (1/6-em)]654 4282
Absent 2379 (15.2) 428 (10.0)
Descriptive 8309 (53.1) 2571 (60.0)
Foundational 4354 (27.8) 1093 (25.5)
Complex 612 (3.9) 190 (4.5)

In summary, we established consistency between rubric users with an interrater reliability investigation and showed utility of the rubric in a variety of contexts (i.e., Organic Chemistry 1 and Organic Chemistry 2 courses), with a variety of reaction mechanisms including varied nucleophile types and reaction families, and with two variations of the constructed response prompt.

Case study – using the rubric in teaching practice

In this section, we offer a case study (Woodside, 2010; Patton, 2015; Yin, 2018) on how this rubric could be used within the context of a course in place of a conventional “implications for instructors” section. Our goal is to demonstrate how assessment results from the rubric can formatively and iteratively inform learning and instruction by developing reflective educators (Henderson et al., 2011).

For this case study, we chose an Organic Chemistry 1 course that was coordinated among two instructors in Fall 2021. Assessment data were collected from all sections of the course, including from sections taught by both instructors, at three time points (i.e., Time Points 1 through 3) throughout the semester, occurring after instruction on examination material but before an examination that assessed learning related to nucleophiles and reaction mechanisms (see Fig. 2 for timeline). The course textbook for Fall 2021 was Klein's Organic Chemistry, 4th edn (2021). The textbook takes a traditional approach to topic order based on functional groups (e.g., alkyl halides, alkenes, alkynes, alcohols, and ethers in Organic Chemistry 1) after initial skill development in chemical reactivity and mechanisms; in the textbook chapter presenting mechanisms, students learn to draw a mechanism and four characteristic arrow-pushing motifs: nucleophilic attack, loss of a leaving group, proton transfer, and carbocation rearrangement. The reaction motifs structure is carried out throughout the course (into Organic Chemistry 2, as well) and aims to help learners develop more expert-like thinking by understanding the isolated pieces of a mechanism (Ferguson and Bodner, 2008; Grove et al., 2012a) and integrate this knowledge into the whole (Galloway et al., 2018) as new functional groups and mechanisms are studied.


image file: d2rp00184e-f2.tif
Fig. 2 Timeline of the case study: course content, examinations, and assessment time points.

Time Point 1 is a measure of where students are when they first encounter nucleophiles and reaction mechanisms; chronologically it is the third of five total course examinations (see Fig. 2). Table 10 shows the distribution of levels using the nucleophile rubric for three different reactions of alkenes (i.e., a halogenation, hydration, and halohydrin formation) that were assessed at this time point. A participating student would have only seen and written about one of these reactions. For the course context and students, the Foundational level is considered as the level of satisfactory understanding; in other words, this is the level of explanation sophistication desired to be exhibited by students. As found in our analysis, the bulk of the responses are generally at the Descriptive level with fewer responses at the target Foundational level. We should note that students received extra credit for completing the assessment; and thus, motivation may have played a role in how and why they responded. Overall, the majority of students identified the nucleophile (i.e., at the Descriptive level or above) in each reaction, which is key.

Table 10 Distribution of levels of explanation sophistication for case study time point 1: reactions of alkenes
Level Halogenation Hydration Halohydrin formation Total
n 276 273 277 826
Absent 31 (11.2) 37 (13.6) 27 (9.7) 95 (11.5)
Descriptive 120 (43.5) 167 (61.2) 153 (55.2) 440 (53.3)
Foundational 116 (42.0) 57 (20.9) 93 (33.6) 266 (32.2)
Complex 9 (3.3) 12 (4.3) 4 (1.5) 25 (3.0)

Time Point 2 is characteristic wherein students have had the opportunity to reflect on the topics of the previous examination and potentially hone their skills relating to nucleophiles and reaction mechanisms as more reaction chemistry is taught; this time point occurs prior to the fourth of five course examinations (see Fig. 2). For this assessment, students wrote about one of the four nucleophilic substitution reactions in Table 11 and one of the two reactions of an alkene or alkyne in Table 12; the letter designation A or B after the substitution reaction type designates two different variations of the reaction. Each student wrote about a total of two reaction mechanisms; these were randomly presented to each student, and thus, students working at the same time on completing the extra credit assessment likely saw different reaction mechanisms or the same reaction mechanisms but in a different order.

Table 11 Distribution of levels of explanation sophistication for case study time point 2: substitution reactionsa
Level SN1A SN1B SN2A SN2B Total
a Letter A and B designations refer to two different variations of each reaction.
n 206 203 206 207 822
Absent 9 (4.4) 18 (8.9) 13 (6.3) 18 (8.7) 58 (7.0)
Descriptive 124 (60.2) 122 (60.1) 135 (65.5) 136 (65.7) 517 (62.9)
Foundational 68 (33.0) 54 (26.6) 51 (24.8) 52 (25.1) 225 (27.4)
Complex 5 (2.4) 9 (4.4) 7 (3.4) 1 (0.5) 22 (2.7)

Table 12 Distribution of levels of explanation sophistication for case study time point 2: reactions with alkenes and alkynes
Level Alkene hydrohalogenation Alkyne alkylation
n 410 411
Absent 31 (7.6) 54 (13.1)
Descriptive 174 (42.4) 251 (61.1)
Foundational 162 (39.5) 102 (24.8)
Complex 43 (10.5) 4 (1.0)

For all reactions at this time point, the majority of responses are again at the Descriptive level demonstrating that students can identify the nucleophile with nucleophilic behavior described at a surface-level. Generally, there are fewer responses at the Absent level and more at the Foundational level when compared with Time Point 1 (Table 10) while noting that there is some variation depending on the exact reaction mechanism; for example, there is a larger number of responses in the Complex level for the alkene hydrohalogenation reaction which has aspects of halogenation reactions from Time Point 1. While we resist a causal explanation, per the assessment measures at Time Point 2, students’ level of explanation for nucleophiles appears to be becoming more sophisticated since Time Point 1; this is somewhat plausible given broader metrics of causality including temporality, time on task, etc. However, we still note that, on average, students are not at the designated Foundational level goal.

Time Point 3 is the final measure of students’ understanding of nucleophiles in reaction mechanisms at the end of the Organic Chemistry 1 course, occurring directly before the last of the five course examinations (see Fig. 2). Here, students were asked to write about one of the three substitution reactions with alcohols (see Table 13), and one of the four reactions of an aldehyde or ketone to form an alcohol (see Table 14). Again, these distributions show that students are not at the target Foundational level. In fact, students may be at the same level of understanding for previously assessed mechanisms; for example, the reaction of an alcohol with hydrobromic acid (HBr) and the alkyne alkylation reaction are analogous to an SN2 reaction with respect to the nucleophilic attack by a lone-pair-containing, anionic nucleophile and has a similar distribution to those reactions. Additionally, these reactions assess understanding of reduction reactions (see Table 14) for the first time. The reduction reactions with lithium aluminum hydride (LiAlH4) or sodium borohydride (NaBH4) have a larger number of students at the Absent level than previously observed with other reaction types which may be the result of less time to understand this newly taught reaction type. These hydride reduction reactions contrast Grignard reactions, which the latter are taught analogously, but has higher levels of explanation sophistication with a larger proportion of responses in the Descriptive level.

Table 13 Distribution of levels of explanation sophistication for case study time point 3: substitution reactions with alcohols
Level Using HBr Using PBr3 Using SOCl2 Total
n 262 266 261 789
Absent 20 (7.7) 13 (4.9) 31 (11.9) 64 (8.1)
Descriptive 173 (66.0) 121 (45.5) 150 (57.5) 444 (56.3)
Foundational 64 (24.4) 111 (41.7) 75 (28.7) 250 (31.7)
Complex 5 (1.9) 21 (7.9) 5 (1.9) 31 (3.9)

Table 14 Distribution of levels of explanation sophistication for case study time point 3: reactions of aldehydes and ketones to form alcohols
Level LiAlH4 with aldehyde NaBH4 with ketone Grignard with aldehyde Grignard with ketone Total
n 197 197 200 195 789
Absent 37 (18.8) 50 (25.4) 16 (8.0) 12 (6.2) 115 (14.6)
Descriptive 132 (67.0) 117 (59.4) 162 (81.0) 154 (79.0) 565 (71.6)
Foundational 25 (12.7) 26 (13.2) 19 (9.5) 26 (13.3) 96 (12.2)
Complex 3 (1.5) 4 (2.0) 3 (1.5) 3 (1.5) 13 (1.6)

In summary, these assessment results show that the majority of students are not at the Foundational target level when explaining nucleophiles in the context of reaction mechanisms. One proposition is that students fail to describe or interpret chemical reactions due to insufficient practice with engaging with these rich descriptions in textbooks and course materials (McCollum and Morsch, 2022). As we will later note, there is an opportunity for more assessments throughout the learning experience to emphasize and model the desired learning level. Overall, we do observe a trend that explanations became more sophisticated over the semester-long course, which may be the result of continuing practice in writing about reaction mechanisms.

Reflective teaching practices

From a summative, post-semester reflection, these assessment results are an opportunity to consider bigger changes we might make to the course to better achieve our goal of a majority of students explaining nucleophiles at the Foundational level or higher.

The assessments, as noted, were given as extra credit assignments prior to each of the last three course examinations of the term. While the instructors fully expected students to develop a deep understanding of the reaction mechanisms, our assessment evidence suggests there is room for improvement. An advantage of using a rubric, such as the one reported herein, is that students are able to use the rubric to monitor their learning and instructors are able to use the rubric to assess learning (Panadero and Jonsson, 2013; see users and uses in Dawson, 2017); thus, we need to share the rubric with students as part of the formative assessment and feedback process (see secrecy in Dawson, 2017). Additionally, exemplary responses are key to the learning process. We need to share quality responses for each level of the rubric that fit within the specific assessment item and context of our course (see exemplars in Dawson, 2017); more importantly, we need to communicate our Foundational level goal to students in the classroom and through examples. The rubric in and of itself can act as a centerpiece for feedback discussion. Instructor comments about a learner's writing can aid in students’ explanations (see accompanying feedback information in Dawson, 2017).

From a formative perspective, the rubric and associated assessment data provide a means to address learning in the ongoing context of the course. Having students complete such writing tasks throughout instruction, modeling expected language in instruction (Andrade, 2005), and using such items on for-credit assessment would further emphasize the importance of this type of learning. As a particular example based on the assessment data we presented above, some of the responses at Time Point 1 are at the Absent level; this is a point of reflection for us, the educator, to ensure we note and label parts of reaction mechanisms (e.g., nucleophile, electrophile, leaving group) and mechanistic steps (e.g., nucleophilic attack, loss of leaving group, proton transfer) as they are presented, and used, and repeated, and appear in homework, etc. making sure to explicitly note the nucleophile in every context. We, as practicing chemists, do not need to be this explicit; however, to a learner, an overemphasis can be key. In considering moving from Descriptive to Foundational, educators have an opportunity to again be routine, in ad nauseum, in making the implicit explicit: we should note the most nucleophilic part of the nucleophile (e.g., the atom and lone pair, sigma bond, pi bond) in each reaction mechanism and mechanistic step, as appropriate. Instances of being “overly explicit” are not meant to be made for the entirety of the year-long course; at some point, learning should be expected. However, if we assume that some level of explicitness is made and thus some level of scaffolding is provided to students, our work and the research literature we summarized herein would support the notion that we are not yet doing enough cueing or scaffolding to reach our desired levels of learning. Our goal then is that later in the semester, when we notice more students are at the Foundational level (i.e., Time Point 2), we can better introduce the “next” level of learning and emphasize electrostatic interactions using differences in electronegativity, electron density, and partial charges to model language we would expect at the Complex level.

Implications for teaching practice

Throughout the collection of the data reported in the broader manuscript and the data specifically reported in this case study, students’ only reference for developing an importance of this type of written explanation assessment was in completing the extra credit assignments that had these prompts; if writing about the “how and why?” is desirable, instructors must provide students with opportunities to practice and appropriately evaluate these skills. Thus, we must be intentional about using explanations in our assessments (formative and summative) and correspondingly, using this rubric and other means to clearly communicate our learning expectations (Cooper, 2015; Stowe and Cooper, 2017).

These assessment data provide a means to consider broader teaching, learning, and assessment transformations, and more importantly, conduct quasi-experimental investigations in our course to make decisions about what those transformations could be. For example, the same prompt and reaction mechanism could be used at the beginning of the semester (or when nucleophiles and reaction mechanisms are introduced) and then again at the end of the semester; the rubric could then assist in longitudinal evaluation. To make that example slightly more explicit, a bimolecular nucleophilic substitution reaction could be used to evaluate nucleophile understanding shortly after instruction or near an examination of their understanding; then, towards the end of the term, the same reaction could be used, and the results of the second assessment can be used to make claims about progression of explanation sophistication of nucleophiles. Additionally, modification of teaching strategies could be rooted in analyzing distributions of responses to discern variation in reactions or reaction types. For example, the distribution between the reactions of aldehyde and ketones with hydride reagents compared with Grignard reagents as noted previously; these are similar mechanisms, and yet, distributions varied and thus as instructors, we have an opportunity to use the assessment to talk about those similarities. Additionally, the prompt could be varied throughout the course to be more or less cued, as needed, or to fit the specifications of our expectations and learning objectives of the course. For example, as we have presented in this case study, our expectations were for Organic Chemistry 1 students; expectations for students enrolled in a non-majors organic chemistry course or for a general, organic, and biochemistry course may be different.

Limitations

There are three limitations of note for this work: (i) instructor implementation of the rubric, (ii) homogeneity of the sample, and (iii) limited number of prompt variations.

Instructor implementation

Differences in instructor expectations may yield differing implementations of the rubric when evaluating level of explanation sophistication. We note several differences in this section that emerged during the interrater process and in peer debriefing discussions; examples of such disagreements are:

Authors had contrasting beliefs regarding the strength of a nucleophile. For example, for a reduction reaction, a response stated, “the ketone is reduced by a moderately strong hydrogen nucleophile … there are two enantiomer products because originally the molecule was planar and could be attacked either from on top or from bottom.” This response was classified in the Descriptive level by one author that believed the strength of a nucleophile was out of the scope regarding nucleophilic behavior in reaction mechanisms and another author that considered descriptions about the strength of nucleophiles classified this response in the Foundational level.

Beliefs about the strength of nucleophiles evolve into conceptions about what makes a “good” nucleophile. A response could have stated that the “oxygen acts as a nucleophile and performs a substitution reaction … and attacks the beta-carbon” and thus be classified at the Descriptive level; however, the added language for an epoxidation reaction that says, “hydroxide has a high affinity for hydrogen [on the alcohol] and removes it. The oxygen becomes a good nucleophile and attacks the beta-carbon” could be argued to be of higher understanding of the nucleophile and classified at the Foundational level.

The Foundational level is characterized as explanations that demonstrate understanding that electrons are central to nucleophilic behavior and arrows represent the movement of electrons. For example, consider this partial response to an aldol condensation mechanism:

“The base comes in and grabs an alpha proton to carry out deprotonation of the alpha carbon on the right because it is more stable. The electrons of the carbonyl group move up to the oxygen forming a negative charge and from deprotonation a double bond is formed between the carbonyl carbon and the alpha carbon or rather the alpha and beta carbon. This intermediate now acts as the nucleophile and attacks another ketone molecule at the carbonyl carbon because it has electrophilic character, pushing electrons up onto the oxygen which bears a negative charge.”

This response has language concerning electrons before and after the nucleophilic step, and thus demonstrates sufficient understanding that nucleophilic behavior and mechanistic arrows represent the movement of electrons at the Foundational level. However, the rubric could be applied such that responses must explicitly state the use of electrons in the nucleophilic step to be at the Foundational level; because this response does not make that explicit, the response could be classified at the Descriptive level.

Electrophilic aromatic substitution reactions, for example, resulted in differing applications of the rubric. In a response to an electrophilic aromatic chlorination, a student wrote, “formation of the electrophile occurs because of the nucleophilic attack from the lone pair [from] chlorine to AlCl3.” This could be classified at the Descriptive level or at the Foundational level; we noted in prior work (Dood et al., 2018; Yik et al., 2021) that some students used nucleophile–electrophile interactions to describe the behavior of Lewis bases and acids, respectively.

These examples show how the rubric could be interpreted and applied differently based on one's ideas about nucleophiles. This, however, when made transparent, can catalyze discussions between educators, exemplify the nature of chemistry and science, and clarify course learning objectives. Instructors can have differing opinions rooted in their beliefs and expectations for students; in the end, though, instructors have a responsibility to make those expectations clear to students and consistently evaluate those expectations accordingly.

Sample homogeneity

Responses used in this study were written by students taught by one of four instructors at a single institution that uses a single curriculum. We should note that those instructors routinely coordinate their course sections; thus, an analogous curriculum is enacted in the courses from which data were collected, with students having analogous learning experiences. We therefore caution against blind adoption of the rubric without consideration of the learners, curriculum, and context. Adaptation of the rubric may be necessary in the context of other curricula such as a spiral organic curriculum (Grove et al., 2008), Mechanisms before Reactions curriculum (Flynn and Ogilvie, 2015), or the Organic Chemistry, Life, the Universe and Everything (OCLUE) curriculum (Cooper et al., 2019). We note that these curricula are associated with different reasoning frameworks; for example, the causal mechanistic reasoning framework (Cooper et al., 2016; Crandell et al., 2019, 2020) is associated with OCLUE.

We also note the emergence of discussions related to equity in STEM when assessments require written explanation or other written work; this is particularly important when the language required to be used in the assessment is not the learner's first language. For example, Deng et al. (in press) reported that students who learned English-as-an-additional language demonstrated lower levels of explanation sophistication compared to their English-first language counterparts. At our study site (i.e., the University of South Florida), we are aware that a growing number of learners are refining their English skills (i.e., the language of instruction at our study site) while simultaneously learning the language of organic chemistry, chemistry, science, etc. As we further develop our rubric and understand the utility of using it in our enacted curriculum, we intend to be attentive to the barriers to learning, access, and retention consequences that our assessments may have.

Limited prompt variations

While our work reports the wide utility of the rubric to evaluate the level of explanation sophistication for nucleophiles with nearly 20[thin space (1/6-em)]000 responses, these responses were gathered using two almost identical prompts (see Table 3). These limited prompt variations restrict instructors to rely on two similarly worded, formulaic prompts that focus on describing what is happening and why it is happening on a molecular level. Our decision to modify the Original prompt to the More Cued prompt was based in work to understand students’ use of causal mechanistic reasoning (Cooper et al., 2016; Crandell et al., 2019, 2020). Our prompts have parallel structure: “Consider the mechanism for the [reaction name/type] to form [product]” or “consider the mechanism for the [reaction] between [substrate] and [reagent] to form [product].” We advocate for instructors to use this general form when first applying our reported rubric; such use can provide a foundation to consider more situational variations to match the learning goals and context in which the rubric is used.

Future endeavors need to consider how cued, scaffolded, and other varied prompts are associated with the types and kinds of responses obtained and how explanations might be integrated into a course. In our work, we demonstrate the generalizability of this rubric with two prompt variations (see Table 3). We can envision prompts that further cue students into activating resources about electrostatic interactions or scaffold responses such that students first identify reaction components before describing the how and why of a reaction mechanism. Writing assignments that ask students to explain reaction mechanisms may take the form of formative assessments or through homework. These assignments could also be embedded into larger summative projects; one example could include a literature project at the end of the year-long organic chemistry course sequence where students find a reaction covered in the course and explain the mechanism for the noted transformation; the literature project could have multiple components wherein explanation of a reaction mechanism is only a component to the summative assessment experience.

Implications

In this paper, we report a rubric for evaluating student level of explanation sophistication for a single reaction mechanism concept: nucleophiles. Using the same sophistication levels (i.e., absent, descriptive, foundational, and complex), the overall rubric structure could be applied to other reaction mechanism concepts (e.g., electrophiles, leaving groups, proton transfers). We envision a broad set of rubrics to evaluate explanations of reaction mechanisms with every possible component of a reaction mechanism or holistically (i.e., a single level for the entirety of a response).

Data used in this study were collected over several years and give a broad picture of how students understand and explain nucleophiles throughout the course. At the same time, our data do not show how this understanding is associated with student achievement (e.g., course grades) and development of understanding over time. Studies have shown that a rubric used to evaluate student work on constructed response items was associated with higher student achievement possibly due to operationalizing definitions of achievement that students could understand (Shafer et al., 2001). We do not yet report external measures to demonstrate increase in student understanding (e.g., examination scores) which could serve as an additional metric to evaluate courses and curricula. Furthermore, as mentioned in our Case Study discussion, assessment of understanding in a longitudinal setting would provide a stronger reference point for course evaluation and revision. A longitudinal approach could take form throughout a single-semester course, a year-long course, or across the undergraduate curriculum. In combination with other evaluative criteria, understanding across time may serve as a useful tool for evaluating curricula and teaching practices.

Scoring student written responses is tedious, especially for large enrollment courses such as Organic Chemistry. Computer-assisted predictive scoring models can be used to evaluate written assessments. The speed to quickly evaluate many responses with predictive scoring models makes written responses a viable option for in-class use (e.g., Haudek et al., 2011, 2012; Prevost et al., 2013; Dood et al., 2018, 2020a; Noyes et al., 2020; Yik et al., 2021). While predictive scoring models are becoming more commonplace, these models are specific to single assessment items; previous work (Yik et al., 2021), though, has demonstrated that generalized predictive models can be developed. A limitation, however, of that work is the binary classification for correct or incorrect/non-use of a very specific conceptual model (Yik et al., 2021). In moving toward more meaningful assessment measures, researchers must aim to (a) develop generalized predictive models to eliminate limitations for item-specific measures, and (b) have means to evaluate responses on a continuum or scale, such as using the rubric reported herein.

Conclusion

We report a generalized rubric for level of explanation sophistication for nucleophiles in organic chemistry reaction mechanisms. We characterized this rubric with a set of design elements for a quality rubric. Our results suggest that this rubric is generalizable throughout a year-long organic chemistry course, across multiple prompt types, with different nucleophile types, and in a variety of reaction families and types. We also present a case study where the rubric was used to classify responses which informed reflective practices and actionable items to enhance teaching and learning experiences. This nucleophile-focused rubric is a starting point for other reaction mechanism components (e.g., intermediates or electrophiles) that can be applied in assessing curricula and teaching practices. Future work might involve using the levels of explanation sophistication described in this rubric as categories for a machine learning-based predictive scoring model.

As one student reflected in their explanation of a reaction mechanism, “I realize that if I had a better understanding of why the reactants are interacting it would make the mechanism easier to comprehend instead of just memorizing the mechanism.” Our overarching principal goal is to create environments that catalyze such learning. This begins with clear outcomes as expressed in our reported rubric, use of that rubric to assess learning, and revising and refining teaching practices to promote such learning.

Author contributions

BJY and JRR conceived the project. BJY, AJD, and JRR collected data assisted by DCR and KBF. BJY, AJD, SJHF, and JRR conceptualized the rubric. BJY cleaned, prepared, and analyzed the data. BJY and JRR wrote the manuscript. All authors read, edited, and approved the final manuscript.

Conflicts of interest

There are no conflicts to declare.

Appendix 1: nucleophiles rubric

Operationalized rubric for levels of explanation sophistication for nucleophiles with exemplary responses for a bimolecular nucleophilic substitution reaction
Level Description of level Key features of level Exemplar
Absent • No response • The nucleophile is not identified “The first step of the SN2 reaction is a nucleophilic attack followed by the loss of a leaving group. The interaction happens at the same time.”
• Non-normative • The nucleophile–electrophile reaction step is missing from the explanation
Descriptive • Describes the nucleophile engaging in bond forming processes • The nucleophilic molecule or atom is identified “The iodine is acting as a leaving group and detaching from the 1-iodopropane. At the same time, the cyanide is attacking the molecule and substituting the leaving group. The reactants and intermediates interact because this is a substitution reaction. Iodine is a good leaving group so therefore this triggers the substitution reaction. Prompting the cyanide to replace it.”
• Simplistic description of bond forming processes • Electrons are not used to describe nucleophilic behavior
• Nucleophilic behavior is described at a surface, atomic level • Bond forming processes are described (e.g., “nucleophile attacks/forms a bond with electrophile”)
Foundational • Nucleophilic behavior is described at a surface, electronic level • Electrons are central to nucleophilic behavior and arrows represent the movement of electrons “The I is a good leaving group and it is leaving while the electron pair on the carbon connected to the triple bond is attacking the carbon that the leaving group is leaving from. They all interact in the same step simultaneously.”
• Explicit features are mentioned • Example descriptors of electrons: sigma electrons/bond, pi electrons/bond, lone pair (and not alkene or double bond)
• Implicit features may be mentioned, but not fully explained • Bond forming processes are described using electrons (e.g., “the lone pair on the nucleophile attacks/forms a bond with electrophile”)
Complex • Describes why the nucleophile is involved in bond forming processes • Implicit electronic features are used to describe nucleophilic behavior “The cyanide is acting as a nucleophile attacking the partially positive alpha carbon kicking off the iodine leaving group, thus forming the product. The electron rich cyanide is attracted to the alpha carbon because of its partial positive charge.”
• Nucleophilic behavior is described at a deeper, electronic level • Examples of electronic properties: electron density, electronegativity, partial charges, molecular orbital descriptions (e.g., HOMO/LUMO)
• Explicit features are used to infer implicit features that are sufficiently explained • Bond forming processes are described using electrons and electronic properties (e.g., “the high electron density on the nucleophile is attracted to the area of low electron density on the electrophile”)

Acknowledgements

We would like to thank all the students who participated in our study. We would like to thank Frankie Costanza (University of South Florida) for providing access to students for data collection. Author BJY would like to thank the Office of Graduate Studies at the University of South Florida for support through a Dissertation Completion Fellowship.

References

  1. Akkuzu N. and Uyulgan M. A., (2016), An epistemological inquiry into organic chemistry education: Exploration of undergraduate students' conceptual understanding of functional groups, Chem. Educ. Res. Pract., 17(1), 36–57 10.1039/C5RP00128E.
  2. Anderson T. L. and Bodner G. M., (2008), What can we do about ‘Parker’? A case study of a good student who didn't ‘get’ organic chemistry, Chem. Educ. Res. Pract., 9(2), 93–101 10.1039/B806223B.
  3. Andrade H. G., (2000), Using rubrics to promote thinking and learning, Educ. Leadersh., 57(5), 13–18.
  4. Andrade H. G., (2005), Teaching with rubrics: The good, the bad, and the ugly, Coll. Teach., 53(1), 27–31 DOI:10.3200/CTCH.53.1.27-31.
  5. Anzovino M. E. and Bretz S. L., (2015), Organic chemistry students' ideas about nucleophiles and electrophiles: The role of charges and mechanisms, Chem. Educ. Res. Pract., 16(4), 797–810 10.1039/C5RP00113G.
  6. Anzovino M. E. and Bretz S. L., (2016), Organic chemistry students' fragmented ideas about the structure and function of nucleophiles and electrophiles: A concept map analysis, Chem. Educ. Res. Pract., 17(4), 1019–1029 10.1039/C6RP00111D.
  7. Bangert-Drowns R. L., Hurley M. M. and Wilkinson B., (2004), The effects of school-based writing-to-learn interventions on academic achievement: A meta-analysis, Rev. Educ. Res., 74(1), 29–58 DOI:10.3102/00346543074001029.
  8. Becker N., Noyes K. and Cooper M., (2016), Characterizing students’ mechanistic reasoning about London dispersion forces, J. Chem. Educ., 93(10), 1713–1724 DOI:10.1021/acs.jchemed.6b00298.
  9. Bell B. and Cowie B., (2001), The characteristics of formative assessment in science education, Sci. Educ., 85(5), 536–553 DOI:10.1002/sce.1022.
  10. Bhattacharyya G., (2013), From source to sink: Mechanistic reasoning using the electron-pushing formalism, J. Chem. Educ., 90(10), 1282–1289 DOI:10.1021/ed300765k.
  11. Bhattacharyya G., (2014), Trials and tribulations: Student approaches and difficulties with proposing mechanisms using the electron-pushing formalism, Chem. Educ. Res. Pract., 15(4), 594–609 10.1039/C3RP00127J.
  12. Bhattacharyya G. and Bodner G. M., (2005), “It gets me to the product”: How students propose organic mechanisms, J. Chem. Educ., 82(9), 1402–1407 DOI:10.1021/ed082p1402.
  13. Bhattacharyya G. and Harris M. S., (2018), Compromised structures: Verbal descriptions of mechanism diagrams, J. Chem. Educ., 95(3), 366–375 DOI:10.1021/acs.jchemed.7b00157.
  14. Birenbaum M. and Tatsuoka K. K., (1987), Open-ended versus multiple-choice response formats—it does make a difference for diagnostic purposes, Appl. Psychol. Meas., 11(4), 385–395 DOI:10.1177/014662168701100404.
  15. Bodé N. E., Deng J. M. and Flynn A. B., (2019), Getting past the rules and to the WHY: Causal mechanistic arguments when judging the plausibility of organic reaction mechanisms, J. Chem. Educ., 96(6), 1068–1082 DOI:10.1021/acs.jchemed.8b00719.
  16. Boesdorfer S. B., Baldwin E. and Lieberum K. A., (2018), Emphasizing learning: Using standards-based grading in a large nonmajors’ general chemistry survey course, J. Chem. Educ., 95(8), 1291–1300 DOI:10.1021/acs.jchemed.8b00251.
  17. Brookhart S. M., (2018), Appropriate criteria: Key to effective rubrics, Front. Educ., 3, 22 DOI:10.3389/feduc.2018.00022.
  18. Brookhart S. M. and Chen F., (2015), The quality and effectiveness of descriptive rubrics, Educ. Rev., 67(3), 343–368 DOI:10.1080/00131911.2014.929565.
  19. Cartrette D. P. and Mayo P. M., (2011), Students' understanding of acids/bases in organic chemistry contexts, Chem. Educ. Res. Pract., 12(1), 29–39 10.1039/C1RP90005F.
  20. Caspari I., Kranz D. and Graulich N., (2018), Resolving the complexity of organic chemistry students' reasoning through the lens of a mechanistic framework, Chem. Educ. Res. Pract., 19(4), 1117–1141 10.1039/C8RP00131F.
  21. Collins H., (2011), Language and practice, Soc. Stud. Sci., 41(2), 271–300 DOI:10.1177/0306312711399665.
  22. Connor M. C., Glass B. H. and Shultz G. V., (2021), Development of the NMR Lexical Representational Competence (NMR-LRC) instrument as a formative assessment of lexical ability in 1H NMR spectroscopy, J. Chem. Educ., 98(9), 2786–2798 DOI:10.1021/acs.jchemed.1c00332.
  23. Cooper M. M., (2015), Why ask why? J. Chem. Educ., 92(8), 1273–1279 DOI:10.1021/acs.jchemed.5b00203.
  24. Cooper M. M. and Stowe R. L., (2018), Chemistry education research—from personal empiricism to evidence, theory, and informed practice, Chem. Rev., 118(12), 6053–6087 DOI:10.1021/acs.chemrev.8b00020.
  25. Cooper M. M., Kouyoumdjian H. and Underwood S. M., (2016), Investigating students’ reasoning about acid–base reactions, J. Chem. Educ., 93(10), 1703–1712 DOI:10.1021/acs.jchemed.6b00417.
  26. Cooper M. M., Stowe R. L., Crandell O. M. and Klymkowsky M. W., (2019), Organic Chemistry, Life, the Universe and Everything (OCLUE): A transformed organic chemistry curriculum, J. Chem. Educ., 96(9), 1858–1872 DOI:10.1021/acs.jchemed.9b00401.
  27. Crandell O. M., Kouyoumdjian H., Underwood S. M. and Cooper M. M., (2019), Reasoning about reactions in organic chemistry: Starting it in general chemistry, J. Chem. Educ., 96(2), 213–226 DOI:10.1021/acs.jchemed.8b00784.
  28. Crandell O. M., Lockhart M. A. and Cooper M. M., (2020), Arrows on the page are not a good gauge: Evidence for the importance of causal mechanistic explanations about nucleophilic substitution in organic chemistry, J. Chem. Educ., 97(2), 313–327 DOI:10.1021/acs.jchemed.9b00815.
  29. Cruz-Ramírez de Arellano D. and Towns M. H., (2014), Students' understanding of alkyl halide reactions in undergraduate organic chemistry, Chem. Educ. Res. Pract., 15(4), 501–515 10.1039/C3RP00089C.
  30. Daniel K. L., (2018), Towards a framework for representational competence in science education, Cham: Springer International Publishing DOI:10.1007/978-3-319-89945-9.
  31. Dawson P., (2017), Assessment rubrics: Towards clearer and more replicable design, research and practice, Assess. Eval. High. Educ., 42(3), 347–360 DOI:10.1080/02602938.2015.1111294.
  32. DeCocq V. and Bhattacharyya G., (2019), TMI (Too much information)! Effects of given information on organic chemistry students’ approaches to solving mechanism tasks, Chem. Educ. Res. Pract., 20(1), 213–228 10.1039/C8RP00214B.
  33. DeFever R. S., Bruce H. and Bhattacharyya G., (2015), Mental rolodexing: Senior chemistry majors’ understanding of chemical and physical properties, J. Chem. Educ., 92(3), 415–426 DOI:10.1021/ed500360g.
  34. Deng J. M. and Flynn A. B., (2021), Reasoning, granularity, and comparisons in students’ arguments on two organic chemistry items, Chem. Educ. Res. Pract., 22(3), 749–771 10.1039/D0RP00320D.
  35. Deng J. M., Rahmani M. and Flynn A. B., (in press), The role of language in students’ justifications of scientific phenomena, Int. J. Sci. Educ., advance online publication DOI:10.1080/09500693.2022.2114299.
  36. Diegelman-Parente A., (2011), The use of mastery learning with competency-based grading in an organic chemistry course, J. Coll. Sci. Teach., 40(5), 50–58.
  37. Domin D. S., Al-Masum M. and Mensah J., (2008), Students’ categorizations of organic compounds, Chem. Educ. Res. Pract., 9(2), 114–121 10.1039/B806226A.
  38. Dood A. J. and Watts F. M., (2022), Mechanistic reasoning in organic chemistry: A scoping review of how students describe and explain mechanisms in the chemistry education research literature, J. Chem. Educ., 99(8), 2864–2876 DOI:10.1021/acs.jchemed.2c00313.
  39. Dood A. J., Fields K. B. and Raker J. R., (2018), Using lexical analysis to predict Lewis acid–base model use in responses to an acid–base proton-transfer reaction, J. Chem. Educ., 95(8), 1267–1275 DOI:10.1021/acs.jchemed.8b00177.
  40. Dood A. J., Fields K. B., Cruz-Ramírez de Arellano D. and Raker J. R., (2019), Development and evaluation of a Lewis acid–base tutorial for use in postsecondary organic chemistry courses, Can. J. Chem., 97(10), 711–721 DOI:10.1139/cjc-2018-0479.
  41. Dood A. J., Dood J. C., Cruz-Ramírez de Arellano D., Fields K. B. and Raker J. R., (2020a), Analyzing explanations of substitution reactions using lexical analysis and logistic regression techniques, Chem. Educ. Res. Pract., 21(1), 267–286 10.1039/C9RP00148D.
  42. Dood A. J., Dood J. C., Cruz-Ramírez de Arellano D., Fields K. B. and Raker J. R., (2020b), Using the research literature to develop an adaptive intervention to improve student explanations of an SN1 reaction mechanism, J. Chem. Educ., 97(10), 3551–3562 DOI:10.1021/acs.jchemed.0c00569.
  43. Ferguson R. and Bodner G. M., (2008), Making sense of the arrow-pushing formalism among chemistry majors enrolled in organic chemistry, Chem. Educ. Res. Pract., 9(2), 102–113 10.1039/B806225K.
  44. Flynn A. B. and Featherstone R. B., (2017), Language of mechanisms: Exam analysis reveals students' strengths, strategies, and errors when using the electron-pushing formalism (curved arrows) in new reactions, Chem. Educ. Res. Pract., 18(1), 64–77 10.1039/C6RP00126B.
  45. Flynn A. B. and Ogilvie W. W., (2015), Mechanisms before reactions: A mechanistic approach to the organic chemistry curriculum based on patterns of electron flow, J. Chem. Educ., 92(5), 803–810 DOI:10.1021/ed500284d.
  46. Galloway K. R., Stoyanovich C. and Flynn A. B., (2017), Students’ interpretations of mechanistic language in organic chemistry before learning reactions, Chem. Educ. Res. Pract., 18(2), 353–374 10.1039/C6RP00231E.
  47. Galloway K. R., Leung M. W. and Flynn A. B., (2018), A comparison of how undergraduates, graduate students, and professors organize organic chemistry reactions, J. Chem. Educ., 95(3), 355–365 DOI:10.1021/acs.jchemed.7b00743.
  48. Galloway K. R., Leung M. W. and Flynn A. B., (2019), Patterns of reactions: A card sort task to investigate students’ organization of organic chemistry reactions, Chem. Educ. Res. Pract., 20(1), 30–52 10.1039/C8RP00120K.
  49. Goodwin W., (2003), Explanation in organic chemistry, Ann. NY Acad. Sci., 988(1), 141–153 DOI:10.1111/j.1749-6632.2003.tb06093.x.
  50. Goodwin W. M., (2008), Structural formulas and explanation in organic chemistry, Found. Chem., 10(2), 117–127 DOI:10.1007/s10698-007-9033-2.
  51. Graulich N., (2015), The tip of the iceberg in organic chemistry classes: How do students deal with the invisible? Chem. Educ. Res. Pract., 16(1), 9–21 10.1039/C4RP00165F.
  52. Graulich N. and Bhattacharyya G., (2017), Investigating students' similarity judgments in organic chemistry, Chem. Educ. Res. Pract., 18(4), 774–784 10.1039/C7RP00055C.
  53. Graulich N., Hedtrich S. and Harzenetter R., (2019), Explicit versus implicit similarity – exploring relational conceptual understanding in organic chemistry, Chem. Educ. Res. Pract., 20(4), 924–936 10.1039/C9RP00054B.
  54. Grove N. P., Hershberger J. W. and Bretz S. L., (2008), Impact of a spiral organic curriculum on student attrition and learning, Chem. Educ. Res. Pract., 9(2), 157–162 10.1039/B806232N.
  55. Grove N. P., Cooper M. M. and Cox E. L., (2012a), Does mechanistic thinking improve student success in organic chemistry? J. Chem. Educ., 89(7), 850–853 DOI:10.1021/ed200394d.
  56. Grove N. P., Cooper M. M. and Rush K. M., (2012b), Decorating with arrows: Toward the development of representational competence in organic chemistry, J. Chem. Educ., 89(7), 844–849 DOI:10.1021/ed2003934.
  57. Haudek K. C., Kaplan J. J., Knight J., Long T., Merrill J., Munn A., Nehm R., Smith M. and Urban-Lurain M., (2011), Harnessing technology to improve formative assessment of student conceptions in STEM: Forging a national network, CBE Life Sci. Educ., 10(2), 149–155 DOI:10.1187/cbe.11-03-0019.
  58. Haudek K. C., Prevost L. B., Moscarella R. A., Merrill J. and Urban-Lurain M., (2012), What are they thinking? Automated analysis of student writing about acid–base chemistry in introductory biology, CBE Life Sci. Educ., 11(3), 283–293 DOI:10.1187/cbe.11-08-0084.
  59. Henderson C., Beach A. and Finkelstein N., (2011), Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature, J. Res. Sci. Teach., 48(8), 952–984 DOI:10.1002/tea.20439.
  60. Howitz W. J., McKnelly K. J. and Link R. D., (2021), Developing and implementing a specifications grading system in an organic chemistry laboratory course, J. Chem. Educ., 98(2), 385–394 DOI:10.1021/acs.jchemed.0c00450.
  61. Klein D. R., (2017), Organic chemistry, Hoboken, NJ: John Wiley & Sons, Inc.
  62. Klein D. R., (2021), Organic chemistry, Hoboken, NJ: John Wiley & Sons, Inc.
  63. Kraft A., Strickland A. M. and Bhattacharyya G., (2010), Reasonable reasoning: Multi-variate problem-solving in organic chemistry, Chem. Educ. Res. Pract., 11(4), 281–292 10.1039/C0RP90003F.
  64. Lapierre K. R. and Flynn A. B., (2020), An online categorization task to investigate changes in students' interpretations of organic chemistry reactions, J. Res. Sci. Teach., 57(1), 87–111 DOI:10.1002/tea.21586.
  65. Lincoln Y. S. and Guba E. G., (1985), Naturalistic inquiry, Beverly Hills, CA: Sage.
  66. Machamer P., Darden L. and Craver C. F., (2000), Thinking about mechanisms, Philos. Sci., 67(1), 1–25 DOI:10.1086/392759.
  67. Martin L. J., (2019), Introducing components of specifications grading to a General Chemistry I course, Enhancing retention in introductory chemistry courses: Teaching practices and assessments, American Chemical Society, ch. 7, vol. 1330, pp. 105–119 DOI:10.1021/bk-2019-1330.ch007.
  68. McClary L. and Talanquer V., (2011), College chemistry students' mental models of acids and acid strength, J. Res. Sci. Teach., 48(4), 396–413 DOI:10.1002/tea.20407.
  69. McCollum B. and Morsch L., (2022), Reading and relationships in organic chemistry, in Reading across the disciplines, Manarin K. (ed.), Bloomington, IN: Indiana University Press, pp. 166–183.
  70. McHorney C. A. and Tarlov A. R., (1995), Individual-patient monitoring in clinical practice: Are available health status surveys adequate? Qual. Life Res., 4(4), 293–307.
  71. National Research Council, (2012), A framework for K-12 science education: Practices, crosscutting concepts, and core ideas, Washington, DC: The National Academies Press DOI:10.17226/13362.
  72. Nedungadi S. and Brown C. E., (2021), Thinking like an electron: Concepts pertinent to developing proficiency in organic reaction mechanisms, Chem. Teach. Int. Best Pract. Chem. Educ., 3(1), 9–17 DOI:10.1515/cti-2021-0009.
  73. Nedungadi S., Paek S. H. and Brown C. E., (2020), Utilizing Rasch analysis to establish the psychometric properties of a concept inventory on concepts important for developing proficiency in organic reaction mechanisms, Chem. Teach. Int. Best Pract. Chem. Educ., 2(2), 20190004 DOI:10.1515/cti-2019-0004.
  74. NGSS Lead States, (2013), Next Generation Science Standards: For States, by States, Washington, DC: The National Academies Press DOI:10.17226/18290.
  75. Nilson L. B., (2015), Specifications grading: Restoring rigor, motivating students, and saving faculty time, Sterling, VA: Stylus Publishing.
  76. Noyes K., McKay R. L., Neumann M., Haudek K. C. and Cooper M. M., (2020), Developing computer resources to automate analysis of students’ explanations of London dispersion forces, J. Chem. Educ., 97(11), 3923–3936 DOI:10.1021/acs.jchemed.0c00445.
  77. Noyes K., Carlson C. G., Stoltzfus J. R., Schwarz C. V., Long T. M. and Cooper M. M., (2022), A deep look into designing a task and coding scheme through the lens of causal mechanistic reasoning, J. Chem. Educ., 99(2), 874–885 DOI:10.1021/acs.jchemed.9b00455.
  78. Panadero E. and Jonsson A., (2013), The use of scoring rubrics for formative assessment purposes revisited: A review, Educ. Res. Rev., 9, 129–144 DOI:10.1016/j.edurev.2013.01.002.
  79. Patton M. Q., (2015), Qualitative research & evaluation methods, Thousand Oaks, CA: Sage Publications.
  80. Petterson M. N., Watts F. M., Snyder-White E. P., Archer S. R., Shultz G. V. and Finkenstaedt-Quinn S. A., (2020), Eliciting student thinking about acid–base reactions via app and paper–pencil based problem solving, Chem. Educ. Res. Pract., 21(3), 878–892 10.1039/C9RP00260J.
  81. Prevost L. B., Haudek K. C., Henry E. N., Berry M. C. and Urban-Lurain M., (2013), Automated text analysis facilitates using written formative assessments for Just-in-Time teaching in large enrollment courses, Presented at the 2013 ASEE Annual Conference & Exposition, pp. 1–15 DOI:10.18260/1-2-19250.
  82. Putica K. and Trivic D. D., (2016), Cognitive apprenticeship as a vehicle for enhancing the understanding and functionalization of organic chemistry knowledge, Chem. Educ. Res. Pract., 17(1), 172–196 10.1039/C5RP00179J.
  83. Raker J. R., Yik B. J. and Dood A. J., (2023), Development of a generalizable framework for machine learning-based evaluation of written explanations of reaction mechanisms from the postsecondary organic chemistry curriculum, in Student reasoning in organic chemistry: Research advances and evidence-based instructional practices, Graulich N. and Shultz G. V. (ed.), The Royal Society of Chemistry.
  84. Reynolds J. A., Thaiss C., Katkin W. and Thompson R. J., (2012), Writing-to-learn in undergraduate science education: A community-based, conceptually driven approach, CBE Life Sci. Educ., 11(1), 17–25 DOI:10.1187/cbe.11-08-0064.
  85. Rivard L. O. P., (1994), A review of writing to learn in science: Implications for practice and research, J. Res. Sci. Teach., 31(9), 969–983 DOI:10.1002/tea.3660310910.
  86. Russ R. S., Scherr R. E., Hammer D. and Mikeska J., (2008), Recognizing mechanistic reasoning in student scientific inquiry: A framework for discourse analysis developed from philosophy of science, Sci. Educ., 92(3), 499–525 DOI:10.1002/sce.20264.
  87. Sevian H. and Talanquer V., (2014), Rethinking chemistry: A learning progression on chemical thinking, Chem. Educ. Res. Pract., 15(1), 10–23 10.1039/C3RP00111C.
  88. Shafer W. D., Swanson G., Bene N. and Newberry G., (2001), Effects of teacher knowledge of rubrics on student achievement in four content areas, Appl. Meas. Educ., 14(2), 151–170 DOI:10.1207/S15324818AME1402_3.
  89. Šimkovic M. and Träuble B., (2019), Robustness of statistical methods when measure is affected by ceiling and/or floor effect, PLoS One, 14(8), e0220889 DOI:10.1371/journal.pone.0220889.
  90. Solomons T. W. G., Fryhle C. B. and Snyder S. A., (2016), Organic chemistry, Hoboken, NJ: John Wiley & Sons, Inc.
  91. Stowe R. L. and Cooper M. M., (2017), Practicing what we preach: Assessing “critical thinking” in organic chemistry, J. Chem. Educ., 94(12), 1852–1859 DOI:10.1021/acs.jchemed.7b00335.
  92. Strickland A. M., Kraft A. and Bhattacharyya G., (2010), What happens when representations fail to represent? Graduate students’ mental models of organic chemistry diagrams, Chem. Educ. Res. Pract., 11(4), 293–301 10.1039/C0RP90009E.
  93. Talanquer V., (2018), Exploring mechanistic reasoning in chemistry, in Science education research and practice in Asia-Pacific and beyond, Yeo J., Teo T. W. and Tang K.-S. (ed.), Singapore: Springer Singapore, pp. 39–52 DOI:10.1007/978-981-10-5149-4_3.
  94. Underwood S. M., Posey L. A., Herrington D. G., Carmel J. H. and Cooper M. M., (2018), Adapting assessment tasks to support three-dimensional learning, J. Chem. Educ., 95(2), 207–217 DOI:10.1021/acs.jchemed.7b00645.
  95. Voorhees R. A., (2001), Competency-based learning models: A necessary future, New Dir. Inst. Res., 2001(110), 5–13 DOI:10.1002/ir.7.
  96. Watts F. M., Schmidt-McCormack J. A., Wilhelm C. A., Karlin A., Sattar A., Thompson B. C., Gere A. R. and Shultz G. V., (2020), What students write about when students write about mechanisms: Analysis of features present in students’ written descriptions of an organic reaction mechanism, Chem. Educ. Res. Pract., 21(4), 1148–1172 10.1039/C9RP00185A.
  97. Watts F. M., Park G. Y., Petterson M. N. and Shultz G. V., (2022), Considering alternative reaction mechanisms: Students’ use of multiple representations to reason about mechanisms for a writing-to-learn assignment, Chem. Educ. Res. Pract., 23(2), 486–507 10.1039/D1RP00301A.
  98. Weinrich M. L. and Sevian H., (2017), Capturing students’ abstraction while solving organic reaction mechanism problems across a semester, Chem. Educ. Res. Pract., 18(1), 169–190 10.1039/C6RP00120C.
  99. Weinrich M. L. and Talanquer V., (2016), Mapping students' modes of reasoning when thinking about chemical reactions used to make a desired product, Chem. Educ. Res. Pract., 17(2), 394–406 10.1039/C5RP00208G.
  100. Wolf K. and Stevens E., (2007), The role of rubrics in advancing and assessing student learning, J. Effect. Teach., 7(1), 3–14.
  101. Woodside A. G., (2010), Case study research: Theory, methods, practice, Bingley, UK: Emerald Group Publishing.
  102. Xue D. and Stains M., (2020), Exploring students’ understanding of resonance and its relationship to instruction, J. Chem. Educ., 97(4), 894–902 DOI:10.1021/acs.jchemed.0c00066.
  103. Yan F. and Talanquer V., (2015), Students’ ideas about how and why chemical reactions happen: Mapping the conceptual landscape, Int. J. Sci. Educ., 37(18), 3066–3092 DOI:10.1080/09500693.2015.1121414.
  104. Yik B. J., Dood A. J., Cruz-Ramírez de Arellano D., Fields K. B. and Raker J. R., (2021), Development of a machine learning-based tool to evaluate correct Lewis acid–base model use in written responses to open-ended formative assessment items, Chem. Educ. Res. Pract., 22(4), 866–885 10.1039/D1RP00111F.
  105. Yin R. K., (2018), Case study research and applications: Design and methods, Los Angeles, CA: Sage Publications.

Footnote

† Electronic supplementary information (ESI) available. See DOI: https://doi.org/10.1039/d2rp00184e
This journal is © The Royal Society of Chemistry 2023