Patterns of reactions: a card sort task to investigate students’ organization of organic chemistry reactions
Kelli R.
Galloway
,
Min Wah
Leung
and
Alison B.
Flynn
*
Department of Chemistry & Biomolecular Sciences, University of Ottawa, Ottawa, Ontario, Canada. E-mail: alison.flynn@uottawa.ca
First published on 16th July 2018
Abstract
Research has shown that within a traditional organic chemistry curriculum, organic chemistry students struggle to develop deep conceptual understanding of reactions and attribute little meaning to the electron-pushing formalism. At the University of Ottawa, a new curriculum was developed for organic chemistry in which students are taught the language of the electron-pushing formalism prior to learning about specific reactions. Reactions are then organized by governing pattern of mechanism rather than by functional group and are taught in a gradient of complexity. To investigate how students are making connections across reactions within the new curriculum, a card sort task was developed. The card sort task consisted of 25 cards, each depicting the reactants and solvent for a reaction taught during the two-semester organic chemistry sequence. The first part of the task asked participants to sort 15 of 25 cards into categories. Then, participants were given the 10 remaining cards to incorporate into categories with the previous 15. Participants were asked to explain the characteristics of each category and their sorting process. Students (N = 16) in an organic chemistry course were interviewed while enrolled in the second semester course. We analyzed the students’ sorts based on which cards were sorted frequently together, the underlying characteristics used to form the categories, and the participants’ sorting processes. Participants created categories based on different levels of interpreting the reactions on the cards, with levels ranging from recognizing identical structural features to identifying similar types of mechanisms. Based on this study, if we want students to develop mechanistic thinking, we think students need to be more explicitly directed to the patterns present in organic reaction mechanisms and given opportunities to uncover and identify patterns on their own, during both summative and formative assessments.
Introduction
Traditionally, organic chemistry is taught by progressing through the reactivity of functional groups. Early work in organic chemistry involved identifying and analyzing functional groups; thus, organizing the curriculum by functional groups made sense at the time. Teaching through functional groups necessitates teaching multiple reaction mechanisms within a unit because a single functional group can react in different ways with different species. In addition, a relatively simple functional group can participate in complex and competing reaction mechanisms. However, research in the teaching and learning of organic chemistry from a functional group approach has revealed many challenges that students have with the electron-pushing formalism (Bhattacharyya and Bodner, 2005; Ferguson and Bodner, 2008; Kraft et al., 2010; Grove et al., 2012a, 2012b) and in using and interpreting organic chemistry concepts (Anderson and Bodner, 2007; Grove and Bretz, 2012; Anzovino and Bretz, 2015, 2016).In organic chemistry, the ways in which students categorize organic chemistry reactions relates to how they think about organic reactions and how they may use that knowledge in a transfer setting. Identifying patterns and making meaningful connections between new and old information is an essential part of learning. People innately categorize and label information as a means to make sense of their experiences (Gentner and Medina, 1998). Identifying patterns allows for inference of new information and transfer of knowledge to new situations (Gentner and Markman, 1997; Yamauchi, 2005).
Students often view learning organic reactions as a task to memorize every reaction the instructor has ever written on the board (Grove and Bretz, 2012). Memorizing organic reactions creates individual bins of information for students rather than allowing for identifying similarities and patterns across reactions. The need for rote memorization could be reduced or eliminated if students learned to identify underlying patterns of reactions, including the mechanistic patterns. We developed a card sort task to explore how students organize organic reactions, whether with meaningful patterns or individual bins of information.
Patterns of mechanisms curriculum
A transformed organic chemistry curriculum was implemented at the University of Ottawa (uOttawa) in 2012 to address the aforementioned learning challenges (Fig. 1) (Flynn and Ogilvie, 2015). The goal of this transformed curriculum has been to provide students with opportunities to learn organic reactions in a systematic way without the need for extensive memorization. If students were better able to master the skills of identifying the roles of the reactive species and analyzing their interactions, then students should more easily able to apply those principles to predicting and explaining unknown reactions. The need to memorize all the reactions covered in class becomes unnecessary because students have a model to think through new and unknown reactions, reducing their cognitive load in the process. Our investigations into the efficacy of the curriculum have so far focused on how students are using and interpreting part of organic chemistry's language—the electron-pushing formalism. This current study investigated how students organize their thinking about organic chemistry reactions using a card sort task. | ||
| Fig. 1 Overview of uOttawa's organic chemistry curriculum. | ||
Card sort tasks as a research tool
One way to explore how people organize their knowledge is using card sort tasks. When people create categories, they create “bins” or “containers” to hold items that have similar features (Yamauchi, 2005). In learning, categories can be viewed as the knowledge frameworks that students develop as they learn new information. When presented with a new experience or new information, a learner must choose what to do with that information and how to store it (Ausubel, 1968; Kirschner, 2002; Novak, 2002, 2010; Mayer, 2012). How the learner chooses to integrate the new information with the old (or not) relates to the organization of the person's knowledge structures (or lack of structures) (Bransford et al., 2010; DiSessa, 1993; DiSessa et al., 2004; Novak, 2010).Card sort tasks are versatile research tools that give insights into the relationships that people perceive that may not be apparent through typical multiple-choice assessments or think-aloud interviews. These types of categorization tasks have been used in multiple areas of research including psychology, physics, biology, and chemistry. Studies using card sort tasks have investigated differences in problem solving strategies for novices and experts (Chi et al., 1981; Smith, 1992; Snyder, 2000; Lin and Singh, 2009; Mason and Singh, 2011, 2016; Smith et al., 2013; Krieter et al., 2016), knowledge organization (Snyder, 2000; Ke et al., 2005; Rottman et al., 2012), organization of representations (Kozma and Russell, 1997; Domin et al., 2008; Stains and Talanquer, 2008; Irby et al., 2016), and teachers’ beliefs (Harwood et al., 2006; Herrington et al., 2011; Aydin and Boz, 2013; Aydin et al., 2014).
The studies using card sort tasks to investigate the transition from novice to expert through problem solving show the usefulness of such tasks for eliciting knowledge organization within a specific discipline and differentiating ways of representing problems. For these categorization tasks, each card had a discipline-specific problem, and participants were asked to sort the cards based on the type of solution. Participants were told they did not need to solve the problem and were not given any pen or paper to discourage problem solving. The “novices” have been identified as first-year students and the “experts” have included upper-division students (third or fourth year), graduate students, and professors. These studies have been conducted using one-on-one interviews (Chi et al., 1981; Snyder, 2000) and in-class activities (Lin and Singh, 2009; Mason and Singh, 2011, 2016; Smith et al., 2013; Krieter et al., 2016). In physics, chemistry, and biology, the studies found that the students tended to sort based on surface feature similarities but that experts created categories for problems with deep feature similarities (Chi et al., 1981; Smith, 1992; Snyder, 2000; Lin and Singh, 2009; Mason and Singh, 2011, 2016, Wolf et al., 2012a, 2012b; Smith et al., 2013; Krieter et al., 2016).
Card sort tasks have been successful at soliciting information on how students think about chemistry representations and the features they attend to. Multiple studies have investigated students’ categorizations of various representations of chemical phenomena (Kozma and Russell, 1997; Domin et al., 2008; Stains and Talanquer, 2008; Irby et al., 2016; Graulich and Bhattacharyya, 2017). Results from these studies have found that students tend to create categories for representations of a similar type (e.g., all graph, all animation or all macroscopic or all microscopic) (Kozma and Russell, 1997; Irby et al., 2016) and use more explicit features to sort the representations (Stains and Talanquer, 2008; Graulich and Bhattacharyya, 2017).
In addition to studying how beginning students classify chemistry representations, the majority of the studies in chemistry also investigated how experts (upper division, graduate students, or professors) would complete the same tasks. A common conclusion across all of these studies was that gaining expertise in chemistry was not linear or continuous. The features that students paid attention to and the decisions they made with those features changed over time or varied with expertise. We concentrated our literature analysis on the findings with beginning chemistry students because the current article focuses on a single sample of chemistry students. We want to understand how students start their knowledge organization to later understand how they develop.
Theoretical framework
Our research was guided by an integration of Information Processing Theory (IPT) with the theory of Meaningful Learning and Human Constructivism (Schunk, 2012). Our understanding of IPT is informed by Johnstone (Johnstone and Selepeng, 2001; Johnstone, 2006), Mayer (2012), and Sweller (Sweller, 1994; Sweller and Chandler, 1994). In the IPT model, people use their five senses to collect information during an experience both consciously and subconsciously. This information is first held in the working memory. If a connection is made between the new information and the long-term memory, then the new information is moved on to be stored in the long-term memory. If no connection is made, then the new information is lost. The modern interpretation of IPT recognizes that humans do not learn robotically, and therefore is deepened by connections with other theories of learning (Schunk, 2012).The learning process described by the IPT directly connects to Ausubel and Novak's theory of Meaningful Learning and Human Constructivism (Ausubel, 1968; Novak, 1993, 2010; Bretz, 2001; Schunk, 2012). Ausubel gave three necessary conditions for meaningful learning to occur (Ausubel, 1968). First, the learner must have relevant prior knowledge. The prior knowledge corresponds to the long-term memory and how knowledge is stored and organized. Second, the new material must be presented in a meaningful way. This condition corresponds to how the learner receives the new information and how it is held in the working memory. The learner could have a well-organized knowledge structure and be able to hold more information in the working memory due to “chunking” and consolidating information. But, if the learner tries to hold too many things in the working memory at once, then the new information either may not be considered in the working memory or may not seem relevant. Finally, the learner has to choose to integrate the new information into the prior knowledge in a non-arbitrary way. Thus, the new information is connected into the long-term memory. A learner can make new connections any time the choice is made to incorporate new information into the long-term memory. These new connections are only meaningful when the learner chooses to make a relevant, non-arbitrary connection (Ausubel, 1968; Novak, 2010).
We conducted this study to investigate how students organize organic chemistry reactions. In a connected study, we explored how organic chemistry graduate students and professors who teach and/or conduct research in organic chemistry approach the same reactions (Galloway et al., 2018). The features that students identify and use to classify and talk about reactions relates to the prior knowledge that is activated when a student sees the reaction. Examining how students think through and organize organic chemistry reactions could reveal how students are making connections across the reactions within the organic chemistry course as well as how meaningful those connections are.
Research questions
To investigate how students organize their knowledge of organic chemistry reactions, we designed a study to explore two research questions: (1) how do students categorize organic chemistry reactions in a patterns of mechanisms curriculum? and (2) What features do students attend to when categorizing the reactions?Methods
Card sort task development
We designed a card sort task to explore the aforementioned research questions. The main decisions in designing the card sort task were to choose which reactions to include on the cards and the layout of the reactions. The options for the layout included: the complete mechanism with reactants, intermediates, products, and electron-pushing arrows; the key mechanistic step with reactants, products, and electron-pushing arrows; reactants and products with no electron-pushing arrows; or reactants only. We decided to include details that would be present on related exam questions: reactants, solvents, and the reaction arrow. We wanted to know how students would analyze the cards without the product present.We brainstormed a list of reactions to include on the cards from the categories of mechanisms taught in Organic Chemistry I (OCI) and Organic Chemistry II (OCII). These reactions were adapted from course materials, previous course exams, and previous research in our group (Flynn and Featherstone, 2017; Galloway et al., 2017). The first list totalled 63 reactions. These reactions were characterized by their type of mechanism, the type and number of functional groups present, which semester the mechanism was taught, and other features such as whether: the reaction was inter- or intramolecular, charge was explicitly present, the key atoms were implicitly or explicitly shown, stereochemical information was explicitly shown, and the reaction would be a familiar or unfamiliar reaction to OCII students. The 63 reactions were narrowed down to 32 reactions by selecting reactions with a variety of characteristics without having too many reactions in the same mechanistic category. For examples of reactions and the selection characteristics, see Table 7, Appendix 1. Then, we sorted the reactions into categories to identify different ways they could be sorted. We narrowed the list to 25 reactions by removing reactions that were repetitive or already had multiple reactions representing a mechanistic category. Each card is labelled with an upper-case letter for identification purposes. A list of the 25 reactions is included in Appendix 2.
The list of 25 reactions was sent to two organic chemistry professors to receive expert feedback. The professors were asked to evaluate the reactions based on their feasibility and whether the presentation of the reactions was similar to a publication or student exam. Based on the expert feedback, revisions were made to the layout of the reactions (e.g., solvent to be under reaction arrow and labelling an acid as a catalyst).
Data collection
We conducted a pilot study prior to the full study (Appendix 3). Based on the results from the pilot study, we revised the interview protocol for the card sort tasks and carried out practice interviews with undergraduate students in our research group. For the first task, the participant was asked to sort only 15 of the 25 cards into categories to aid at decreasing the cognitive load of sorting all 25 cards at once. For the second task, the participant was given the remaining 10 cards to sort into their categories of the previous 15 cards (Stains and Talanquer, 2008). Participants were told they could incorporate the new cards into their existing groups, rearrange the cards, and/or make new categories as long as all 25 cards were sorted. For each task, participants were asked to describe their categories and the reasons for sorting that way. The interviewer asked the participants to name to the categories created during Task 1 and then to re-evaluate the category names in Task 2 and name any new categories. Participants were given dry erase pens, pen and paper, a textbook, and model kit to use if they wanted.Before recruiting volunteers, we modified our ethics approval protocol to increase the compensation from a $5 (pilot study compensation) to a $10 Starbucks gift card. Once the modification was approved, students from two sections of OCII (one English and one French) were invited to participate in an interview. An announcement was made in class, and a handout was posted to the course management site with information about how the students could participate and a link to an online consent form. Of the 26 students who volunteered, 16 students scheduled interviews and participated in the study. The interviews took place over a five-week period, from the end of week 4 through week 9 of the semester. At this point in the course, students had been taught reactions of σ electrophiles. During the weeks of data collection, instruction was focused on spectroscopy, and students were beginning to learn more carbonyl chemistry. Concurrently, organic chemistry graduate students and professors who teach and/or conduct research in organic chemistry were recruited to participate in an interview. The results from those interviews are discussed in a separate publication (Galloway et al., 2018).
The length of the interviews ranged from twenty-seven minutes to an hour and ten minutes with an average interview length of fifty-one minutes. The interviews were audio and video recorded to capture an accurate account of what the students said and their card sorting process. The interviews were conducted in English. French students were encouraged to use language they were comfortable with for terms they did not know in English (e.g., azote for nitrogen).
Of the sixteen OCII student participants, there were six males and ten females. Eleven of the students were in their second year of university, and five were in their third year. For their first language, half of the sample indicated English, one fourth indicated French, and one fourth indicated Other. The sample represented a variety of ethnicities, which is representative of the university and population of the students enrolled in OCII. Course and exam grades were not collected for these students.
Data analysis
To answer the first research question of how the students categorize organic chemistry reactions, we calculated descriptive statistics for the categories, visualized the categories, and qualitatively analyzed the students’ category descriptions for Task 1 and Task 2. To explore the second research question, we continued our qualitative analysis of the specific features the students used to create their categories.Participants’ categories were listed in an Excel spreadsheet where each row corresponded to a participant and each column listed the letters corresponding to the cards in a single category (Table 1). If participants chose to create a category for reactions they did not know, those cards were split up into individual categories to not assume that the participant saw similarities in those reactions. Descriptive statistics were calculated for the average number of categories and average number of cards per category as one way to explore how the students sorted the reaction cards.
| Participant | Task 1 | ||||||
|---|---|---|---|---|---|---|---|
| Category 1 | Category 2 | Category 3 | Category 4 | Category 5 | Category 6 | Category 7 | |
| Madison | DEN | IKO | CGLM | ABHJ | F | ||
| Ryan | AGJ | FIKO | BE | HN | CL | D | M |
| Caden | DKO | FIM | HN | L | CG | AJ | BE |
To visualize the relationship between participants’ categories simultaneously, we used Gephi—a data visualization tool typically used for social network analysis (Bastian et al., 2009). Gephi creates a network of lines between the cards to depict the connections the participants made as they sorted the cards into categories and other relationships between categories (described below). To prepare the data for analysis in Gephi, we first counted the frequency that each card was paired with another card and then mapped the pairs in Gephi. This procedure was carried out for Task 1 and Task 2 of the interview card sort task.
Students’ descriptions of categories and explanations for their sorting processes was analyzed qualitatively. The interviews were transcribed verbatim. Using NVivo 11 for Mac, we began open coding the interviews allowing for the codes to come from the students’ descriptions and explanations (Saldaña, 2013; Corbin and Strauss, 2015). The codes characterized how the students created categories, the features the students identified when describing their categories, and their explanations for their sorting processes. These codes were refined using constant comparative analysis to compare within participants, across participants, within codes, and across codes (Corbin and Strauss, 2015). Throughout the process, memoing was used to organize ideas about the findings (Charmaz, 2001, 2006). After brief analysis of the interviews as a whole, we analyzed the participants’ sorts for Task 1 separately from Task 2. Then, we used the emerging analytical categories from Task 1 to analyze Task 2, adding new codes as new ideas arose. We also analyzed the participants’ approach to Task 2, whether they saw it as a separate task with a clean slate or a continuation of their categories from Task 1.
Establishing trustworthiness
We worked to establish trustworthiness in our study using naturalistic inquiry guidelines (Lincoln and Guba, 1985). We used peer debriefing and negative case analysis to develop our credibility. During peer debriefing, the first author engaged in conversations with two other researchers who had less knowledge of the project. The researchers discussed the emerging findings and possible interpretations of the data. The dialogue gave the researcher opportunities to realize any biases that could influence the development of the conclusions. For negative case analysis, we looked for disconfirming cases and revised the emerging findings when negative evidence was found. One example of negative case analysis could be related to a potential claim that students who make a larger number of categories are using only surface features (i.e., structure, property). Zachary is a negative case for this finding because although he created more than 10 categories, a few of his categories were based on type of mechanism, a few for type of reaction, and a few on type of structure. The correlation between number of categories and type of categories created is not strong. To support the transferability of our research, we provided a thick description of the participants and the context of the study above, and we provide supporting evidence for our conclusions in the form of student quotes.Results and discussion
The descriptive statistics for the number of categories and cards per category for both tasks are shown in Table 2. For Task 1, the number of categories students created ranged from 4 to 13 with an average of 6.7. For Task 2, average number of categories students created was 8.6 with a range of 5 to 15. In general, students created more categories for Task 2 with a similar number of cards per category for both tasks.| Categories | Cards per category | |||||
|---|---|---|---|---|---|---|
| Mean (Stdev) | Min. | Max. | Mean (Stdev) | Min. | Max. | |
| Task 1 | 7.7 (2.2) | 4 | 13 | 2.8 (1.2) | 1 | 9 |
| Task 2 | 8.8 (2.8) | 5 | 15 | 2.9 (1.0) | 1 | 9 |
We will first discuss the students’ overall reaction sorts with the visualizations with Gephi. Then, we will discuss the results from the qualitative analysis of the students’ descriptions of their categories.
Visualizing the data with Gephi
The Gephi map for Task 1 shows an overall picture of how the students sorted the first fifteen reaction cards (Fig. 2). In Gephi, each letter corresponds to a reaction card and is called a “node”; lines between the nodes are called “edges”. The thickness and shade of the edge corresponds to the number of participants who sorted the two cards into the same category. A thicker and darker edge indicates more students put the card pair in the same group whereas a light coloured, thin edge represents a card pair rarely sorted into the same category. From the Gephi map, we see the more frequently sorted card pairs were EN, FI, DK, KO, AG, and AJ (Appendix 4 and Table 8). | ||
| Fig. 2 Gephi map for Task 1 for N = 16 OCII students. For reference, the reaction cards can be found in Appendix 2. | ||
We used the network statistics calculated within Gephi to format the map. The size of each node signifies the node's betweenness centrality—a measure of how often one node is travelled through on the shortest path between two other nodes (Freeman, 1977). In network analysis, nodes with high betweenness centrality act as bridges or junctions linking communities to other communities. For example, in our data, the node for reaction H is the largest node on the map showing that it has the highest betweenness centrality and acts as a bridge between other nodes such as M to D or C to F. In our context, the high betweenness centrality for card H suggests that it was placed in a variety of different categories by the students. Conversely, node N is one of the smallest nodes indicating that students consistently placed it in the same category.
Also, the nodes are coloured into modularity communities, i.e., natural divisions within the network (Blondel et al., 2008). The strength of the divisions for Task 1 was calculated to be 0.27. With a small modularity value, these divisions should be interpreted cautiously, but comparing the modularity communities with the edge thickness supports the identification of the frequent cards pairs. For instance, reactions AGJ (alkene nucleophiles reacting with electrophiles) are included in the community with reactions CLM (electrophilic aromatic substitution and one acid–base reaction). The edges between CLM are not as thick as those between AGJ.
The Gephi map for Task 2 is shown in Fig. 3. Node H continued to be the largest node, acting as a bridge between different communities in the network. The modularity was again low (0.28), but a cautious interpretation shows that cards AGJ have been set apart from the aromatic substitution reactions. Using edge thickness, this map shows that the card pairs that were sorted together with high frequency for Task 2 included the same card pairs from Task 1 plus NQ, EU, CS, CV, TX, and KP (Appendix 4 and Table 9).
 | ||
| Fig. 3 Gephi map for Task 2 for N = 16 OCII students. | ||
Comparing the Gephi maps between Task 1 and Task 2, we see a greater number of card pairs sorted in high frequency in Task 2 than Task 1. Perhaps this increase is because there were more reactions to sort together in Task 2, or because the students saw more similarities among the reactions given a larger pool to sort. Card C was not frequently sorted with other reactions in Task 1 but was frequently sorted with cards S and V in Task 2. The complex web of edges shown in the Gephi map for Task 2 seems to indicate that the student participants did not create reaction categories in similar ways. For example, card H has an edge to every other reaction except for card J indicating that card H was sorted with many different cards by participants and was rarely placed with another card more than once. Perhaps students could not decide how to sort card H.
With Gephi, we only see how the students sorted the reactions and not why they sorted how they did. We can hypothesize and apply our expert chemist lens to interpret the frequent categories, but qualitative analysis is needed to explore the students’ actual reasoning for creating the reaction categories.
Reasons for creating categories
During the interviews, students were asked to describe their categories and explain why they chose to sort the reaction cards in that way. The criteria that students used for creating categories suggested four levels of interpreting the reactions (Fig. 4). The levels were for creating a category based on: identical structural features, properties of structure, type of reaction, and type of mechanism (Table 3). Through the qualitative analysis, we also observed that all students created categories based on reactivity but used different criteria to do. | ||
| Fig. 4 Four main levels of interpreting reaction cards were observed when students created categories. | ||
| Level of interpretation | Description | OCII |
|---|---|---|
| Type of mechanism | Reading reaction for molecular-level interpretation for the process of how the reaction occurs | “Yeah, so these (FIKOTVY) were ketones that acted as electrophiles and on the other if the oxygen part of the ketone or aldehyde is the thing that initiated the reaction, it acts as a nucleophile (BDRX).” Madison |
| Type of reaction | Interpreting structure to think about reactivity and how species interact | “[T]his one (AGM) is an addition of halogen.” Hannah |
| Properties of structure | Some interpretation of structure to identify common characteristics | “[T]hese ones (DEN) all have like pretty strong leaving groups. And they all have stereochemistry to them.” Riley |
| Identical structure | Surface level sorting with no interpretation of structure | “All the [benzene groups] that didn't have either uh a carbonyl of something nitrogen or uh acid I put it in (EGP) and then the ones with carbonyls on the substituents I put as one category (KTV).” Abigail |
The four observed levels of interpretation are depicted as concentric circles to indicate that those in the innermost circle interpret less of the structure on the cards with increasing interpretation and reading of the structure moving outward (Fig. 4). Students who created categories for identical structural features did not mention the properties of those structures, the reactivity, or any mechanistic characteristics as they described their categories. Students who created categories for type of mechanism saw the structural features, the properties of those structures, and how the reaction would proceed and went further to use the movement of electrons and electron density to identify similar features of the reactions. Each level is discussed below.
Zachary created the majority of his categories based upon either having a similar mechanism or similar reaction, but he created one category for having the same structural features. He paired cards FI together because:
Zachary: Like they look similar and they both have like uh like bulky both of these are like bulky solvent things so I left them and yeah I don't know what else.
Zachary used the structural features of the reactions to create a category for FI. Then he named the category based upon the structure of the anhydride. He said:
Zachary: I just looked at the starting material, so I called it that kind of reaction because I don't know actually happens.
Because of Zachary's lack of knowledge of the type of reaction that occurred, he looked only at the structural features of the reactants to create the category.
Ava created all her categories based upon having identical structural features. She sorted CM together for the “NH2 groups” and L alone because it was “just an aromatic.” Ava put KO in one group for having a carbonyl, but then created a category for BDH. She explained:
Ava: It could go either way. It could be the, I don't know whether it would be the alcohol (B) or the double bonded O (D) or the NH (H) or Cl group (H).
Since cards BDH had a carbonyl plus a second functional group, Ava chose to create a separate category for those cards than for those with only a carbonyl.
Connor sorted cards AB into the same category because of the characteristics he identified for the reagents:
Connor: And um like the first I think this is the first one (AB) I choose because I look at the condition and both of them have strong acids. I think sulfuric acid (points at B) is relatively strong and they both have um like a solvent that is electronegative or electro-attractive? That gives electrons, electron donor, electron donor solvent so I grouped this together.
Connor determined that both AB have strong acids and solvents that can donate electrons. He did not mention the effect of the strong acid or the electron-donating solvent on the reaction or even if the reactions would proceed in the same way. Connor created the category for the purpose of having cards where the reactions have a strong acid present and an electron-donating solvent.
Madison created two categories for cards having similar properties. When describing her category for cards IKO, she said:
Madison: O, K, and I are the ones with lithium or sodium in them so they dissociate as soon as they are in the solvent.
Instead of discussing how reactions IKO proceed, Madison characterized the similarities between the cards as having reagents that can dissociate in solution. Both Madison and Riley sorted card DEN together using the same criteria:
Madison: So uh we're learning about like S N 1, E1, E2 and all that so uh that first that stood out was good leaving groups.
Riley: Um these ones all have like pretty strong leaving groups. And they all have um stereochemistry to them. Like the leaving groups have like either in the plane or out of the plane.
Rather than calling DEN substitution or elimination reactions, Madison and Riley labelled the similarity among the cards as all having good leaving groups. Connor, Madison, and Riley attempted to make sense of the information presented on the cards, but their interpretations went only as deep as a description of the physical and chemical characteristics of the structures.
Andrew created each of his nine reaction categories based on having similar types of reactions. Instead of discussing what happened in each reaction, Andrew labelled each reaction category by identifying the type of reaction that he thought would occur. Andrew used his knowledge of the properties of different species in the reaction to help make his decisions. For instance, he called reaction D a Grignard reaction because he recognized the reagents:
Andrew: This (Card D) looks like a Grignard to me because there is um the solvent's called I forget—But yeah it looks like a Grignard there's magnesium and then there's the Grignard solvent and uh a proton donating group.
Andrew thought about the information given for the reaction when deciding how to sort the reactions in to categories. Similar to Andrew, Tyler listed off the names of reactions in lieu of describing the categories they created:
Tyler: Ok so uh here (CL) we are making a substitution on the benzene ring. These two. Um these two (BF) I'm not sure but I guess acid base reaction. Acid base. Acid base. I'm not sure about this one (F). Um for these (DKO) uh making substitution on the carbonyl. Three cases. Uh for here (AGJ) adding to the double bond. Adding to double bond. Uh here (EN) it's an S N 2 (N), one is E2 (E). And those three (HIM) I'm not sure about.
For Tyler, the act of sorting the reactions into categories became a task of identifying the specific type of reaction for each card.
Ryan: So in these three I think I saw an alkene as a nucleophile for all three (AGJ). Um so the electrons from the alkene would attack one of the reagents usually like Cl 2 or whatever and take one. And then another reagent would attack on the carbocation.
Ryan identified similar electron sources and sinks in each of the three reactions and decided they should be in the same category. Ryan also sorted reactions FIK together because of a similar mechanism:
Ryan: And then over here um these ones (FIK) I saw um a carbonyl and a strong nucleophile so I figured it was an addition to a carbonyl for all three or something of that sort.
Interviewer: Ok what's the strong nucleophile on each?
Ryan: Um so over here it'd be the uh carbanion (K). Um over here the solvent (F), this is a weak nucleophile solvent but I think it'd activate that by taking a H [sic] making it a strong nucleophile. Um and then over here (I) not such a strong nucleophile cause it'd react with the solvent I'm not sure. I guess that OH (points at tert-butyl-alcohol).
Again, Ryan identified a source and sink for the electron movement by pointing out the carbonyl as the electrophile in each reaction reacting with a similar nucleophile. Ryan was the only student to sort reaction K with reactions FI and did so with chemistry reasoning. Unlike reactions AGJ, the reactions FIK are not taught together in the curriculum. Reaction K is taught during OCI while reactions FI are taught towards the end of the OCII (which Ryan had not yet been explicitly taught). Using his prior knowledge about electron movement, Ryan made a connection between two reactions with more surface level similarities and a reaction with similar underlying mechanistic characteristics.
Caden considered electron-movement when discussing reaction H. At first, Caden described feeling unsure about reaction H and chose to sort it into a category on its own. Then he began to draw what he thought might happen in the reaction (Fig. 5):
 | ||
| Fig. 5 Caden's use mechanistic arrows to explain reactions H and N helped him decide to sort them into the same category. | ||
Caden: You get that the partial charge (draws above the carbon α to chlorine) and you could move the electrons here to get that negative charge (carbonyl's π bond) so I kind of thought that this oxygen could attack here (carbon α to chlorine) and then the chloride could leave.
Caden did not draw the correct mechanism for the reaction as he has the oxygen of the amide acting as the nucleophile rather than the amine but did consider partial charge and drew electron-pushing arrows from an electron source to an electron sink. Later when discussing reaction N, Caden again drew a mechanism to assist in his description of why he said reaction N was a substitution reaction (Fig. 5). Caden labelled the reaction as substitution and drew the cyanide and iodide almost switching places, but he did not draw partial charges this time. After drawing the mechanism, Caden debated whether the reactions H and N could be in the same category:
Caden: If like if this (H) mechanism is correct, then I don't know, to me it looks pretty different from what this (N) would do, so well I'm not sure actually because it's not that different cause there's still the formation of the charge and then it attacks the positive side. But at the first time, I put them in different categories cause I really wasn't sure about this one (H,) but let's say if that (H) was correct, what I wrote, then I put them in the same category.
Caden acknowledged the formation of the charge and the nucleophile attacking the partially positive carbon implying that he was thinking about electron movement when drawing the curved arrows for both reactions. Once Caden drew mechanisms for reaction H and N, he saw similarities between them and decided they belonged in the same category. Caden's example also shows how students rearranged the reaction cards during the discussion of their categories. Caden sorted in silence, but when he had to speak aloud the categories he created and why, he saw similarities that he had not seen previously.
Approach to Task 2
Participants were given three options for sorting for Task 2: keep existing categories, make new categories, or rearrange categories. Most of the students kept the categories they created in Task 1, sometimes adding a few more categories, and a few students rearranged their categories due to the new cards they were given.Kaitlyn: Um yeah the new ones I don't know I guess when you're hit with new questions you kind of leave behind the old stuff so you want to make new categories as well.
Interviewer: What do you mean by that?
Kaitlyn: Um like it's new task so it's kind of like starting fresh almost. Like obviously I'm not starting fresh. I'm still working with the same cards, but like the mind set almost changes and like you're handed a whole new stack so you kind of change your mindset. I don't know how to describe it. Yeah.
Kaitlyn described how she wanted to approach Task 2 from a new perspective because it's a new task. But, Kaitlyn in fact did not rearrange her categories to have that fresh start for Task 2. She explained why not:
Kaitlyn: Um I guess it was easier just to add it on to together groups as well instead of changing the titles. It's just easier to assume like, ok that was what I came up with that in the beginning, so that's fine. Like I don't have to change it, it's fine. Um so it's easy just to add on.
Kaitlyn decided the task would be easier to just insert the new cards into the categories she had already created instead of finding new characteristics to sort on. She accepted the categories she created for Task 1 were good enough to continue using. Many other students also talked about the advantages of merging the ten new cards into the categories they had already made. Consider Caden's perspective:
Caden: I think I pretty much found the categories that worked for those new cards.
Caden thought the categories he made already worked well and did not see reason to start over. Lily saw an advantage to sorting based on the existing categories:
Lily: Um I suppose a bit because I was trying to see um explain it using the titles first more than actually looking at it. Like I was looking at the titles, oh which one would work best to fit this reaction? Rather than coming up with the title by just looking at the reaction.
Lily realized that she could decrease the effort necessary to sort the reactions by using her previously categories as reference points without having to identify each new reaction. In this way, these students approached Task 2 as if it were a “closed sort” (Rottman et al., 2012; Krieter et al., 2016). The students used the categories they created in Task 1 as the categories by which they would sort the new cards by Task 2 and only created new categories if a new card did not fit.
Abigail decided to rearrange because her categories had become too large. For Task 1, Abigail had created a category for cards that all had benzene in common. For Task 2, she decided to split this group up into separate categories based on the second functional group present on the benzene ring. She explained:
Abigail: And then I realized that the benzene substitution category was huge so um I didn't want to like I wanted it to be more specific, so all the ones that didn't have either uh a carbonyl or something with nitrogen or uh acid I put it in I left it (EGP) and then um the ones with the carbonyls on the substituents I put as one category (KVT). The uh, with a nitrogen I put it on another category with the benzene (CMS) and then this one was just it's a new it's a carboxylic acid (Y). So I put I just put it in a different category because it was a new functional group that I didn't see before.
Abigail saw value in having more categories that were smaller and more specific than fewer categories with more general similarities.
Some students rearranged their previous categories when new cards sparked insight. Connor's rearrangement was initiated by reaction U. Connor explained his thought process:
Connor: Um so I guess my initial thought was there should be like a product that happens but then I look at this (U) and I'm like that's not right. This is just like something put in water. So, I was like I think maybe nothing would happen or maybe it's an elimination. Ok actually like what maybe I'm wrong, but um my initial thought process was like if there is solvent there and a product, would a product of the part, uh initial product, initial molecule then there won't be anything that occurs in the system. So, I was like ok then if this doesn't work, maybe there are other cards shown here that also won't produce anything, and I look at this (J) and I'm like um there's no solvent, usually like a reaction needs solvent to happen, so I'm guessing, so that's why I'm starting to look if there's any reaction that don't happen. Then I, deciding that this four (JLQU) won't happen because either they don't have solvent or they don't have um another molecule another reagent to work with.
As Connor processed reaction U, he wondered if there were similar reactions that could form a new category. Reactions JL were in Connor's “Miscellaneous” group from Task 1, so thinking about the new cards helped him see patterns that he had not previously seen.
Madison made more drastic rearrangements. For Task 1, Madison had sorted based on similar reaction type and similar characteristics of structure, but for Task 2, Madison sorted the reactions into categories solely based on the type of mechanism. Madison explained what initiated the change:
Madison: So I tried looking at them and as soon as I came across this one (S) that fit in three categories I realized that my categories couldn't, weren't ideal. So instead of looking at similarities between the structures, I started looking at similarities between the mechanisms. So, to my understanding these all have similar mechanisms in each category.
When Madison came across card S, it fit into three of her previous categories: benzene, leaving group, and ions that dissociate. To accommodate card S, Madison decided to rearrange her categories to reflect similar types of mechanisms. She described her new perspective:
Madison: Some of them I was a little bit more sure of, so for example leaving groups I knew that leaving groups were a bit part of it. With others for example ones that included the thing above the arrow to dissociate, I knew that wasn't part of the mechanism. It was just the similarity in the compound. So I went through these with like the intention of distributing them over base on the mechanism not their similarity of structures.
Table 4 shows the change in Madison's categories from Task 1 and Task 2. Madison's categories are not perfect (e.g., reaction W's acid chloride would not undergo an SN2, SN1, E2 or E1 reaction), but her sorts show her critical analysis when given examples that do not fit an existing model.
| Madison's rearrangements | |
|---|---|
| Task 1 categories | Task 2 categories |
| Benzene ring reactions (CGLM) | Benzene ring reactions (CLMS) |
| Double bond reactions (ABHJ) | Double bond reactions (AGJP) |
| Good leaving groups (DEN) | Undergoes E1/SN1/E2/SN2 reactions (EHNQUW) |
| Nucleophile dissociates (IKO) | Ketone acts as electrophile (FIKOTVY) |
| Does not belong anywhere else (F) | Ketone acts as nucleophile (BDRX) |
Explanation for sorting process
Despite having a variety reasons to sort the cards into categories, when asked why they chose to sort that way, all students responded for the purposes of reactivity. This finding may seem obvious for students who created categories for type of reaction or mechanism, but even those using structural features to sort the cards did so for the potential inferences they could make regarding reactivity of the structures.Ryan created categories for type of mechanism and type of reaction. When asked how he chose to sort in the way he did, Ryan again discussed the different features of the cards that helped him identify that a similar mechanism would occur. For instance, for reactions FIK, Ryan said:
Ryan: I noticed they all have carbonyls and the reagents included a nucleophile. So something with an electron pair that could donate pretty readily.
Ryan's descriptions suggest that when he looked at the reaction cards, he was interpreting the reaction at a molecular level. He was seeing the structural features and then processing their characteristics and how they would interact together. Then, Ryan was able to discuss his process aloud showing his awareness of his understanding of the interpretation of the symbols on the cards. Sophia, who created all of her categories for type of reaction, said she did so because of the type of reaction that would occur. When asked why she sorted how she did, she responded:
Sophia: Because they were all reactions. So I thought that was a good way to group them.
Sophia felt it was obvious to sort the cards based on the specific type of reactions since each card had a reaction on it.
Connor created categories for properties of structure and said he did so because:
Connor: [U]sually that's how we determine which kind of reaction will happen.
Even though Connor did not talk about similarities based on the reaction that would happen for the categories he created, by sorting based on similar properties of structure, Connor was still thinking about reactivity. Connor's comment suggests that his categories for species with similar properties would allow him to eventually be able to determine what reaction would happen.
Participants who created categories for identical structural features also said they did so because they would react in the same way. Consider how Ava's discussion on how she decided to sort how she did:
Ava: Um trying I guess trying to figure out in my mind try to sort them and figure out like I said before which ones are going to react with which type of mechanisms and which type of reaction.
Ava adhered to the idea that molecules with the same functional group would react in the same way; therefore, sorting the cards by functional group made sense to her because it would tell her something about reactivity.
Discussion of features students attend to while sorting the cards
The features that students attended to while discussing the categories they created were analyzed through the lens of the criteria they gave for creating the categories. Students who created categories focused on the reaction process (similar mechanism or reaction) incorporated some similar features into their descriptions (Table 5). Students in these two levels had two distinguishing factors in the features they discussed. First, only students within the similar mechanism level identified features that indicated thinking about areas of high and low electron density and identifying reactive sites (e.g., partial charges and molecular orbitals). Second, students within the similar reaction type level incorporated many of surface features of the reactions into their descriptions. Within the similar reaction type level, students appeared to have identified structural features to use to figure out how the reaction would proceed or what product would be formed. For example, Andrew said he “identified” a reaction and stated the structural features that he used to do so. Perhaps the structural features that these students identified triggered a memory for a reaction type, suggesting that the students were relying more on their memory to categorize the reactions than thinking about the symbols on the card. Conversely, students who created categories for similar type of mechanism elicited implicit features from the card's symbols and incorporated chemistry knowledge.| Features identified for categories of similar mechanism | Features identified for both similar mechanism and similar reaction | Features identified for categories of similar reaction |
|---|---|---|
| Partial charge | Charge | Acid–base |
| Molecular orbitals | Description of key steps and mechanism | Acid |
| Electrons | Base | |
| Alkene | Description of multiple reaction steps | |
| Carbonyl | Ion dissociation | |
| Electrophilic aromatic substitution | Alcohol | |
| Substitution | Benzene | |
| Nucleophile and/or electrophile | Carboxylic acid | |
| Ketone | ||
| Halogen, chlorine | ||
| FeCl3 | ||
| Metal, Li, Mg | ||
| Intramolecular | ||
| Leaving groups | ||
| Molecular size | ||
| SN1/E1, SN2/E2 | ||
| Grignard | ||
| Number of reaction steps | ||
| Product formation | ||
| Reaction intermediate | ||
| Reagent | ||
| Reducing agent | ||
| Solvent |
Students who created categories within the levels of similar properties of structure and identical structural features only used structural features to sort the cards and did not incorporate aspects of the reaction process into their categorization scheme (Table 6). Within the level of similar properties of structure, students discussed few features of the reactions. Most of the discussion was around the solvent and the reagents (reactant species listed above the reaction arrow). Students who created categories within the level of identical structural features only identified surface features and minimally incorporated features beyond reading the literal letters and numbers on the card.
| Features identified for categories of similar properties of structure | Features identified for both similar property and identical structure | Features identified for categories of identical structure |
|---|---|---|
| Acid | Base | Alcohol |
| Dissociate into ions | Li | Alkene |
| Electrons | Leaving groups | Amine |
| Mg | Solvent | Benzene |
| SN1/E1, SN2/E2 | Carbonyl | |
| Reagents | Ester | |
| Stereochemistry | Halogen, chlorine | |
| Molecule size | ||
| Number of reaction steps | ||
| Reactant |
Conclusions
We developed a card sort task to explore how organic chemistry students organize their knowledge of organic chemistry reactions. Students worked through a first task to sort fifteen organic chemistry reactions into categories based on their own choosing. All students created categories demonstrating the connections they saw among the reactions given to them, but there was diversity in the features that the students used to create their categories. The students drew on their prior knowledge and experiences to identify features they believed to be relevant to sort the reactions into categories. Drawing upon our learning theory framework, how the students created categories was related to the organization of their prior knowledge, working memory capacity, and choices to make connections between the reactions on the cards to their prior knowledge. For some students, the prior knowledge they used to create the categories may have been fragmented which hindered how they were able to make connections between the reactions (e.g., Connor and Abigail). Other students may have had a more organized knowledge structure but perhaps did not make a conscious choice to connect the reactions they were sorting to their prior knowledge.The students’ most prevalent categorization scheme was to create categories for a particular type of reaction. In this way, students were thinking about the process that was being conveyed on the card but did not discuss the underlying patterns of mechanisms. Some students created categories only for structural features whether looking at the properties of the features or sorting based on identical functional groups. Few students used patterns of electron movement to create categories.
For a second task, students were given ten additional cards to add to their sorting of the first fifteen where they could incorporate the cards into existing categories, make new categories, or rearrange their categories. Many students evaluated their original categories to decide whether they were useful or if they needed to be revised based on the new reactions. They sought to identify similar characteristics between the new ten cards and the previous fifteen.
No students created the exact same categories. Gephi maps show that students sorted some cards together more frequently than others, but students had different reasons for sorting the same reactions into categories. The Gephi maps also display the variety in the students’ sorts. Although the students were all taking the same course, they had different interpretations of the same reactions, which also connects back to the different prior knowledge each student brings to the learning experience. There was variation in the number of categories students created as well as the number of cards per category. Students who tended to create more categories created smaller categories and distinguished the reactions in greater detail. Students who created a lesser number of categories had more cards per category and chose to see greater similarity between the reactions whether that was based on functional groups or reaction types. Students who created fewer, more detailed categories could have less organized knowledge structures and view the reactions more as individual silos than connected by underlying patterns. In this study, the students were asked to sort the reactions in any way they would like. The results show the patterns in reactions that students saw on their own and choose to use. Had students been asked to sort the “best way,” for a specific purpose, or even by the type of mechanism, they might have organized the cards in different ways.
Using our findings from the present study as a framework (i.e., four categorization types: structural features, properties, reaction type, and mechanism type), we explored and compared how organic chemistry graduate students and professors who teach and/or conduct research in organic chemistry approach the same reactions (Galloway et al., 2018).
Limitations
Student participants were volunteers from a convenience sample and may not reflect the overall demographics of all the students who take OCII. The range of students’ responses—from structural similarities through mechanistic detail—indicate that not a single kind of student volunteered to participate. The range of responses could point to the diverse students who were interviewed. Furthermore, the interviews were a single snap-shot of students’ organization of organic reactions; they may think about reactions a variety of other ways. Future studies will incorporate the card sort task into the course structure to investigate how the students’ categorization schemes change over time as they learn reactions. The students’ responses were limited by the reactions they were given on the cards. On the cards, we included details that would be present on related exam questions: reactants, solvents, and the reaction arrow. However, the students might have sorted the cards in different ways if also given the products or mechanistic details (i.e., using different representations). Additionally, the students’ interpretation of the external representation of the chemical equations could affect their sorting of the cards (Taber, 2009). Future research could integrate interview or in-class questions for students’ interpretations of what the card depicts. The participants were not asked to make judgments or solve problems based on the categories they created; therefore, we cannot conclude that the categories they created here exactly reflect how they would think about organic reactions in another situation, such as on an exam. Rather, our goal was to explore how students are thinking about organic reactions in an environment with lower stakes than a midterm or final exam. The interview participants were from both English and French sections of OCII and the students came from diverse backgrounds. Future studies could conduct the interviews in the language of the section in which the students are enrolled to explore the different role that language could play in categorizing reactions. The interviews were conducted over a relatively short five-week time span. Students could have experienced growth during that time. Future studies could use the card sort in longitudinal studies to explore how students’ categories change throughout a course or course sequence.Implications for teaching
Our evidence with the card sort task shows that students are finding patterns and similarities in organic reactions but are not always choosing or seeing how to organize the mechanistic patterns as organized by the curriculum, even though these patterns had been explained with some limited practice opportunities. Although similarities in structure can be useful for certain situations, we think that by helping students to see the underlying pattern of mechanism we can provide a foundation for them to develop a strong conceptual understanding of mechanisms without feeling the need to resort to memorization of reactions.The results from this study raised concerns that students may not see, be capable of seeing or value the patterns of mechanisms, but instructors can build in opportunities to help students discover and work with the patterns themselves. One example would be to give students a set of reactions that appear to be different on the surface but with similar mechanisms with the instruction to list everything the reactions have in common such as an alkene reacting with an acid and alkene reaction with a peroxide or an addition to a carbonyl compared to an addition to an imine. Another option is to incorporate closed and open sorts into class activities (Fig. 6). In this way, students have opportunities to become skilled in identifying common underlying features and must work within those bounds to identify the similar characteristics. These kinds of activities can be implemented throughout the course both as formative and summative assessment questions (Fig. 6). Once students are able to identify the patterns on their own and see deep features in common, they can begin to develop a lens to apply in unknown situations.
 | ||
| Fig. 6 Sample assessment questions to help students discover patterns across organic chemistry reactions. | ||
Students need guidance for how to identify useful features and how to use the features to determine useful ways to think about organic chemistry reactions. For example, students could be asked to identify specific useful features such as:
• Where are areas of high and low electron density?
• Are there polar bonds? Are there pi bonds?
• What kind of bond is being formed?
• What is the rate-determining step?
If we point out a feature only once, students are very unlikely to remember every time in the future (Bransford et al., 1989). We must consistently ask students what they see and predict as well as identifying features we know to be important and prompting them to consider why a given feature might be useful. Another classroom example would be to incorporate contrasting cases (Bransford et al., 1989). With contrasting cases, students are given reactions with structural similarities but with different mechanisms (perhaps the first two reactions on the top row of Fig. 6) and asked to identify the differences in the reactions. This activity could help the students become familiar with the unique nuances between different types of mechanisms.
In this study, when students were asked to sort new cards in Task 2 with their categories from Task 1, they chose to make connections between new information and previous information. The students who rearranged their categories did so because they found new connections between the reactions that they had not seen the first time. By creating scaffolding within our courses to help students see connections across seemingly different ideas, we can attempt to safeguard them from seeing each new course unit as a self-contained silo of information.
Implications for research
This research builds on the previous card sort literature in chemistry and science education extending the card beyond discipline-specific problems and type of visualizations to organic reactions. The card sort was a useful tool for starting the conversation with the students about how they think about organic reactions. The format of the card sort tasks allowed for exploration of how students might integrate new information with previous knowledge to help us understand how they organize and categorize organic chemistry knowledge and construct aligned learning opportunities. Longitudinal studies could use the card sort task at multiple time points throughout OCI and OCII to investigate how students’ responses change as they learn organic chemistry and what features they attend to with different levels of knowledge. Future uses with the card sort task could also incorporate cards with mechanisms and products to examine how students view the different information provided as they look for similarities among the reactions.Conflicts of interest
There are no conflicts of interest to declare.Appendix 1: sample list of reactions and their considered characteristics for inclusion in the card sort task
| Reaction | Mechanism | FGs (# FGs) | Inter or intra? | Charge? | Implicit atoms reacting? | Semester | Familiar? | Include? |
|---|---|---|---|---|---|---|---|---|
|
Electrophilic aromatic substitution | Aromatic, acid (2) | Inter | Formal | Yes | 1 | Yes | Yes |
|
π electrophile, no LG | Imine, aromatic, Grignard (3) | Inter | Partial | No | 1 | Yes | Yes |
|
π electrophile, with hidden LG | Aldehyde, alcohol, acid (3) | Intra | Formal | Explicit | 1 | Yes | Yes |
|
π nucleophile, aldol | Ketone, aldehyde, base (3) | Inter | Formal | No | 2 | No | Yes |
|
π nucleophile | Alkene, acid (2) | Inter | Formal | Yes | 1 | Yes | Yes |
|
π nucleophile | Alkene, aromatic, halide (3) | Inter | Partial | No | 1 | Yes | Yes |
|
σ electrophile, E1 | Alkene, alkyl halide (2) | Inter | Partial | No | 2 | Yes | Yes |
|
σ electrophile, SN2 | Amide, amine, alkyl halide (3) | Inter | Partial | No | 2 | No | Yes |
|
π nucleophile, Michael addition | α,β-Unsaturated ketone, hydride (3) | Inter | Yes | 2 | No | No | |
|
σ electrophile, SN2 | Alkyl halide, triphenylphos phine (2) | Inter | Partial | No | 2 | Yes | No |
|
π electrophile | Amide, base (2) | Inter | Formal | No | 2 | No | No |
|
π electrophile, no LG | Aldehyde, phenol, ether, hydride, acid (5) | Inter | Formal | No | 1 | Yes | No |
|
π electrophile? rearrangement | Carbonyl, amine (2) | Intra | Partial | No | 3 | No | No |
|
π nucleophile, no LG | Alkyne, alcohol, acid (3) | Inter | Formal | Yes | 1 | Yes | No |
|
π nucleophile, no LG | Alkene, acid (2) | Inter | Formal | No | 1 | Yes | No |
|
σ electrophile, SN2 | Epoxide, amine, alkyl halide (3) | Inter | Partial | Yes | 2 | No | No |
|
π nucleophile | Alkene, hydride, peroxide, base | Inter | Partial | Yes | 1 | Yes | No |
Appendix 2
Task 1: Sort these 15 cards into categories.
Task 2: Incorporate these additional 10 cards into your sorting of the previous 15. You can keep the same categories, rearrange, or make new as long as all 25 cards are sorted.
Appendix 3: pilot study
Before collecting data, we applied for and received approval from the Office of Research Ethics and Integrity at our university to conduct research with human subjects.We conducted a pilot study with the card sort task to ensure adequate collection data for our intended purpose. For the pilot study, we developed a qualitative interview protocol with two card sort tasks. First, the researcher gave the participant all 25 cards to sort into categories. After the participant was satisfied with the categories, the researcher asked the participant to give a name to the categories created, describe each category, and to explain the sorting process. For the second task, the researcher asked the participant to sort the same 25 cards into new categories and then asked the same follow-up questions. The participants were given dry erase pens (the cards were laminated for longevity), pen and paper, a textbook, and model kit to use throughout the tasks if they wanted. The interviews were audio and video recorded to capture an accurate account of what the participant said and their sorting process. The video camera was set up to capture how the participant interacted with the cards and was not pointed at the face. Participants received a $5 Starbucks gift card as compensation.
Summer research students in the Department of Chemistry & Biomolecular Sciences were recruited to participate in an interview. Pilot interviews were conducted with three students in July and August 2016. Each student had previously completed the organic chemistry course required for his or her program, but only one student had completed an organic chemistry course at the University of Ottawa. The interviews were transcribed verbatim, and the videos were watched to transcribe the sorting process and what the participants pointed to during the interview. We conducted preliminary qualitative analysis to explore how the card sort tasks elicited the students’ organization of organic chemistry reactions. For the first task, each student was able to identify characteristics to sort the reactions and discuss their reasoning. For the second task, one student gave up and chose not to complete the task. Throughout the interview, all three students exhibited fatigue, and two of the three students expressed overwhelm at the number of cards to sort. As for the data collection methods, the video did allow for adequate capture of the participant's sorting process as intended and did not appear to require changes for the full study.
Appendix 4: frequently sorted card pairs from Task 1
| Task 1 | |
|---|---|
| Cars pairs | Number of students who sorted |
|
11 |
|
9 |
|
8 |
|
8 |
|
8 |
|
7 |
| Task 2 | |
|---|---|
| Cars pairs | Number of students who sorted |
|
10 |
|
9 |
|
9 |
|
9 |
|
8 |
|
8 |
|
7 |
|
7 |
|
7 |
|
7 |
|
7 |
|
7 |
Appendix 5: additional supporting examples
Type of reaction
Isabella created three of her eight categories for reactions of similar types. First, she described sorting together reactions EIKN because they were substitution reactions:Isabella: Alright so these ones (EIKN)… Like these ones (EN) I would think of that one I would think of nucleophile substitution. Sorry I don't know how to say that in English. And these ones (IK) are more like whatever I learned last year. But I know for example like um this one (K) would probably attack the carbonyl and then for me that considered kind of like it wouldn't be a substitute it'd be an addition but like I kind of consider as a substitution because instead of having a two bond O you have like one oxygen with one OH and then one um ethyl added to it. So basically it's kind of the same trend that goes on.
Isabella labeled reactions EN as nucleophilic substitution reactions and reactions IK as substitution at the carbonyl group. In her mind, because all four reactions involved a substitution of some sort, they could go into the same category. Isabella identified BM as acid–base reactions but chose to not discuss further beyond identifying an acid and a base in both reactions. Then, she described the category for reaction AGJ:
Isabella: So that one (AJG) I considered just breaking alkenes. Like a C–C bond.
At first it may seem as if Isabella is thinking about the mechanism for these reactions, but she did not discuss electron movement or mention that the double bonds break because they are nucleophiles. She saw the alkene would break in each reaction and decided they could fit together in the same category.
Kaitlyn's descriptions of the seven categories she created were all based on having a similar reaction. For instance, she chose to sort reactions JI together because she thought they both would form epoxides:
Kaitlyn: Ok so these two (JI) I thought they would make epoxides because I know mCPBA does epoxides (J). This one (I) I'm not too sure about the reaction does but I was kind of thinking SN2 but intramolecular so like it would form an epoxide somehow.
Kaitlyn put J and I in the same category for the purpose of forming the same product and acknowledged that the process for forming that product would be different for both reactions. Kaitlyn also sorted reaction CDHK together because she said each reaction would be adding an alkyl group to the reactant. She started off with DK in the same group:
Kaitlyn: And then these two (DK) I though they would like this one (D) I thought would be a Grignard but then it didn't have a bromine so maybe a chlorine Grignard because I'm familiar with the solvent. And then this one (K) would act similar to it not a Grignard but it would just add the alkyl group.
Later, she reconsidered reaction C:
Kaitlyn: Um oh this one (C) I might add over here (DK) just because they're all the actually n like nevermind wait a sec. Sorry I'm just.
Interviewer: “Oh that's fine. Take your time.
Kaitlyn: Ok well all three of these reactions are similar (DKC).
Interviewer: “Ok how are they similar? What made you decide to move that card?
Kaitlyn: Um because they would all be involved in adding an alkyl group so except this one (D) I can't really see that there is an alkyl group that would be added. It would be like the preparation for a Grignard, it's only, wait, it should be step 1 and 2, oh god, ok, um … I'm just going to go ahead and say that's a Grignard for now. Ok. Those would all be like adding alkyl groups.
Interviewer: And what's the alkyl group, can you point it out to me again that's adding on Card C?
Kaitlyn: “Card C it would be this one (points to CH2 in CH2Cl2).
Kaitlyn struggled with reaction H and eventually she landed on the idea that the alkyl group would add to the reactant as well. So then she added reaction H to the category with CDK:
Kaitlyn: I feel like it'd be some sort of addition reaction with an alkyl group but I'm not sure how it'd work. Like the other ones at least I know like somehow they'd added on but I'm not sure how this one would go. … Like I see that there's an alkyl group and I mean why would there be an alkyl group if it's not going to be added like it's not going to just sit there so like maybe yeah. Maybe I will add it here even though I'm unsure. Um yeah. Yeah I'll just add it there. (Moves H to with CDK)
Andrew identified reaction J as an “epoxidizing” reaction, reaction E and N as either substitution or elimination, reaction D as a Grignard, reaction C as Friedel Crafts Acylation, reaction B an M as “simple acid–base,” reactions AG as substitution on the double bond, reaction F as oxidative cleavage, reaction O as possibly forming an acetal, and reaction H, K, and L as nucleophile addition reactions.
And he labeled reaction O as forming an acetal because of the presence of two ketones:
Andrew: This one (Card O) I had a little bit of trouble with. I looked through and the only thing that caught my eye is something that might possibly be is since there's two ketone groups it might be an acetal somehow and since there's a base you can make a ring like a like a carbocation ring instead of just two acetals by themselves you could actually do something with the double bond to make a ring instead of a long chain.
Even though Andrew incorrectly labeled reaction O, Andrew attempted to pull forth his prior knowledge of acetals to figure out what reaction would occur.
Like Tyler, Sophia also listed off reaction names for each of the categories she created:
Sophia: Ok. So this (EN) is I identified as substitution. Like nucleophilic substitution. Um this one (C) ah I wasn't sure actually. I don't know why I put it here. I was actually not sure what it was. But I think that I was like um hm no I don't think that it is a substitution but I'm not sure for this one. This one (B) is dehydration of an alcohol which I found in a textbook. This one (M) is um acid base. This one (A) is um electrophile addition. This one (K) is acylation. Um so this (J) is epoxide. Uh sulfonation of benzene (L). I don't know what this one (G) is called but it's when the Cl goes here and here. Um these (F and I) I don't know what they are but they are um some kind of ester reaction like they both have this group so I think that they'll react. They'll have some similar way of reacting. And then these (DHO) here are ketones and I put this one (O) as addition of organometallic reagent and then these two (HD) I don't recognize and I couldn't find them in the textbook but I know they're ketones so I put them with that one (O).
Tyler saw more similarities among the reaction types than Sophia by creating categories with more than one card. Sophia created categories with the majority her categories having only one card. As she identified the type of reaction for each reaction listed, she claimed there be to distinct differences between the types of reactions:
Tyler: Like if it was a different kind of reaction it's like put it in a different kind of category.
Sophia's thinking of the reactions within specific types limited her visibility of similarities between the reactions.
Properties of structure
Riley also used the presence of stereochemistry as a similarity between the cards. Continuing the characteristics related to substitution and elimination reactions, Riley described her category for cards BJL:Riley: These ones (B, J, L) are all strong bulky bases so they're gonna be either SN2 and E2 reactions.
While Riley says the reactions would be either SN2 or E2, she gives the category the name “strong, bulky bases” suggesting the main criteria for creating the category was for the characteristic of the reagent rather than the reaction that would proceed. When asked to elaborate on how the cards fit together, she pointed at sulfuric acid (BL) and mCPBA (J) as the strong, bulky bases in common on each card.
Identical structural features
Abigail created the majority of her categories based upon the reactions having the same functional groups. She sorted HM together “because they have an amine,” AG together for the alkene, and BDKO together for the carbonyl. Abigail did acknowledge that carbonyls could react in different ways but chose to focus on the presence of the carbonyl:Abigail: I mean they might react differently, but I'm just, I classified them because I feel like um the other functional groups that they have, like this one (B) has an alcohol and then this one (K) has like a benzene you know, but I feel like the carbonyl will be the main focus I think.
Miscellaneous
Some students would choose criteria to sort the reaction cards that would leave some cards unsorted. These leftover cards were put into either “Miscellaneous” or “Unknown” categories. For some students, there would only be one leftover card:Sophia: Um this one (C) ah I wasn't sure actually. I don't know why I put it here. I was actually not sure what it was. But I think that I was like um hm no I don't think that it is a substitution but I'm not sure for this one.
Madison: And F I wasn't sure at all.
Both Sophia and Madison sorted the other 14 cards into categories but each had a card leftover that they were not sure how to sort. Sophia did not remember reaction C and so was not sure how to categorize it while Madison identified two ketones on reaction F but was not sure how the functional group interacted with the other reactant hindering her from sorting F with other cards.
Three students created categories that resulted in five to seven leftover cards. Connor had chosen to create categories based upon the structural features having similar properties. After he had sorted the cards with similar characteristics, he was left with cards that did not fit his chosen pattern. He explained:
Connor: Um this I don't see any pattern in any of this (CFGHJLN). Maybe I'm wrong. Probably I'm wrong. […] Yeah so I don't see any of the uh can be put together in any of the groups or within the existing groups that I've created. So this is just kind of standalone conditions I guess.
Once Connor selected characteristics to sort on, he had seven cards that did not fit within his sorting criteria. Instead of looking for additional characteristics among these seven leftover cards, Connor left them in a “Miscellaneous” category.
Isabella and Ella also had leftover cards they chose not to sort into categories. Both students outwardly admitted they did not know where to start with interpreting the reactions:
Isabella: Um as for this (CDFLO) I truly just I made like a list of things that I don't know.
Ella: This is the pile of things I have no idea what's going on. (FGJKLO) Yeah. No idea, like, how even to start with the reaction.
Isabella and Ella had sorted their other categories by similar type of reaction, so when they got to reactions that were unfamiliar to them, they were not sure how to sort them. Unlike the students who sorted by identical structural features, Isabella and Ella did not try to identify any similar characteristics based on their lack of knowledge of how the reactions would proceed.
Approach to Task 2
Ryan and Tyler used the same categories for both tasks without creating any new categories. Ryan explained his process:Ryan: I figured it worked pretty well in terms of categorizing for the first one and now that I had ideas for categories I'd just kind go through them again and see if they fit under one of the categories or if I have to make a new one and they all seem to fit except for this one but based on the reagents I feel like that's the only possible reaction so then I put it under that category.
Ryan thought his categories from Task 1 had served him well, so he chose to see if the new cards would fit under his existing categories. He held each new card up to his existing categories until he found a good fit and kept to his previous mindset of sorting by similar mechanism or reaction type.
Andrew also saw an advantage to sorting based on the existing categories:
Andrew: I feel like this time was a lot um easier I want to say because I've already had these predefined groups I can start looking for similarities as opposed to differences between the cards I can look for similarities between this new card and then the groups I have laid out.
Andrew recognized that he could now look for how the new cards fit within the existing categories rather than trying to distinguish the new cards from each other.
Rearranging
In a similar way to Abigail, as Isabella sorted for Task 2, she created a category that she eventually broke into two smaller categories. She said:Abigail: I had like pretty much everything with nucleophile but like I kind of decided to separate it because there was too much in one category.
Isabella had reactions E, N, F, and R together because she saw they all had a nucleophile. Then she decided to separate E and N from F an R based on their different types of reactions. Like Abigail, Isabella chose to categorize based on more specific features resulting in fewer, smaller categories.
For Task 1, Sophia had put C and L in their own individual categories. For Task 2, she combined C and L with reaction S. She described:
Sophia: Um then these (CS to L) were um I put I changed this to just um substitution on a aromatic group. Because I think like all of these like this one (L) will add SO3, this one (S) will add NH2, this one (C) will add um chlorine to the group depending on like whichever spot.
Seeing reaction S led Sophia to create a category for substitutions on an aromatic group where she had not seen a similarity between C and L previously.
Acknowledgements
This work was supported by the University of Ottawa. The authors thank the Flynn Research Group for their feedback on the research.References
- Anderson T. L. and Bodner G. M., (2007), What can we do about ‘Parker’? A case study of a good student who didn’t ‘get’ organic chemistry, Chem. Educ. Res. Pract., 9(2), 93–101.
- Anzovino M. E. and Bretz S. L., (2015), Organic chemistry students’ ideas about nucleophiles and electrophiles: the role of charges and mechanisms, Chem. Educ. Res. Pract., 16(4), 797–810.
- Anzovino M. E. and Bretz S. L., (2016), Organic chemistry students’ fragmented ideas about the structure and function of nucleophiles and electrophiles: A concept map analysis, Chem. Educ. Res. Pract., 17, 1019–1029.
- Ausubel D. P., (1968), Educational Psychology: A Cognitive View, New York, NY: Holt, Rinehart, and Winston, Inc.
- Aydin S. et al., (2014), Examination of the topic-specific nature of pedagogical content knowledge in teaching electrochemical cells and nuclear reactions, Chem. Educ. Res. Pract., 658(15), 658–674.
- Aydin S. and Boz Y., (2013), The nature of integration among PCK components: A case study of two experienced chemistry teachers, Chem. Educ. Res. Pract., 14(4), 615–624.
- Bastian M., Heymann S. and Jacomy M., (2009), Gephi: An Open Source Software for Exploring and Manipulating Networks, Third International AAAI Conference on Weblogs and Social Media, pp. 361–362.
- Bhattacharyya G. and Bodner G. M., (2005), ‘It gets me to the product’: How students propose organic mechanisms, J. Chem. Educ., 82(9), 1402–1407.
- Blondel V. D. et al., (2008), Fast unfolding of communities in large networks, J. Stat. Mech.: Theory Exp., DOI:10.1088/1742-5468/2008/10/P10008.
- Bransford J. D. et al., (1989), New approach to instruction: Because wisdom can’t be told, in Vosniadou S. and Ortony A. (ed.), Similarity and Analogical Reasoning, Cambridge: Cambridge University Press, pp. 470–497.
- Bransford J. D., Brown A. L. and Cocking R. R., (2010), How people learn: Brain, mind, experience, school, How people learn: Brain, mind, experience, school, Washington, D.C.: National Academies Press.
- Bretz S. L., (2001), Novak's theory of education: Human constructivism and meaningful learning, J. Chem. Educ., 78, 1107.
- Charmaz K., (2001), Qualitative interviewing and grounded theory analysis, in Gubrium J. F. and Holstein J. A. (ed.), Handbook of interview research, Thousand Oaks, CA: SAGE Publications, pp. 675–692.
- Charmaz, K., (2006), Constructing grounded theory: a practical guide through qualitative analysis, London: Sage Publications.
- Chi M. T. H., Feltovich P. J. and Glaser R., (1981), Categorization and representation of physics problems by experts and novices, Cognit. Sci., 5, 121–152.
- Corbin J. and Strauss A., (2015), Basics of qualitative research, Thousand Oaks, CA: SAGE Publications, Inc.
- DiSessa A. A., (1993), Toward an epistemology of physics, Cognit. Instr., 10(2–3), 105–225.
- DiSessa A. A., Gillespie N. M. and Esterly J. B., (2004), Coherence versus fragmentation in the development of the concept of force, Cognit. Sci., 28(6), 843–900.
- Domin D. S., Al-Masum M. and Mensah J., (2008), Students’ categorizations of organic compounds, Chem. Educ. Res. Pract., 9, 114–121 Search PubMed.
- Ferguson R. and Bodner G. M., (2008), Making sense of the arrow-pushing formalism among chemistry majors enrolled in organic chemistry, Chem. Educ. Res. Pract., 9, 102–113.
- Flynn A. B. and Featherstone R. B., (2017), Language of mechanisms: exam analysis reveals students’ strengths, strategies, and errors when using the electron-pushing formalism (curved arrows) in new reactions, Chem. Educ. Res. Pract., 18(1), 64–77.
- Flynn A. B. and Ogilvie W. W., (2015), Mechanisms before Reactions: A Mechanistic Approach to the Organic Chemistry Curriculum Based on Patterns of Electron Flow, J. Chem. Educ., 92(5), 803–810.
- Freeman L. C., (1977), A set of measures of centrality based on betweenness, Sociometry, 40(1), 35–41.
- Galloway K. R., Leung M. W. and Flynn A. B., (2018), Comparison of how undergraduates, graduate students, and professors organize organic chemistry reactions, J. Chem. Educ., 95(3), 355–365.
- Galloway K. R., Stoyanovich C. and Flynn A. B., (2017), Students’ Understanding of Mechanistic Language Prior to Learning Organic Reactions, Chem. Educ. Res. Pract., 18(2), 353–374.
- Gentner D. and Markman A. B., (1997), Structure mapping in analogy and similarity, Am. Psychol., 52(1), 45–56.
- Gentner D. and Medina J., (1998), Similarity and the development of rules, Cognition, 65(2), 263–297.
- Graulich N. and Bhattacharyya G., (2017), Investigating students’ similarity judgments in organic chemistry, Chem. Educ. Res. Pract., 18, 774–784.
- Grove N. P. and Bretz S. L., (2012), A continuum of learning: from rote memorization to meaningful learning in organic chemistry, Chem. Educ. Res. Pract., 13(13), 201–208.
- Grove N. P., Cooper M. M. and Cox E. L., (2012a), Does mechanistic thinking improve student success in organic chemistry? J. Chem. Educ., 89(7), 850–853.
- Grove N. P., Cooper M. M. and Rush K. M., (2012b), Decorating with arrows: Toward the development of representational competence in organic chemistry, J. Chem. Educ., 89(7), 844–849.
- Harwood W. S., Hansen J. and Lotter C., (2006), Measuring teacher beliefs about inquiry: The development of a blended qualitative/quantitative instrument, J. Sci. Educ. Technol., 15(1), 69–79.
- Herrington D. G. et al., (2011), Target inquiry: changing chemistry high school teachers’ classroom practices and knowledge and beliefs about inquiry instruction, Chem. Educ. Res. Pract., 12, 74.
- Irby, S. M. et al., (2016), Use of a card sort task to assess students’ ability to coordinate three levels of representation in chemistry, Chem. Educ. Res. Pract., 17(2), 337–352.
- Johnstone A. H., (2006), Chemical education research in Glasgow in perspective, Chem. Educ. Res. Pract., 7, 49–63.
- Johnstone A. H. and Selepeng D., (2001), A language problem revisisted, Chem. Educ. Res. Pract., 2(2), 19–29.
- Ke J. L., Monk M. and Duschl R., (2005), Learning introductory quantum physics: Sensori-motor experiences and mental models, Int. J. Sci. Educ., 27(13), 1571–1594.
- Kirschner P. A., (2002), Cognitive load theory: implications of cognitive load theory on the design of learning, Learn. Instr., 1–10.
- Kozma R. B. and Russell J., (1997), Multimedia and understanding: Expert and novice responses to different representations of chemical phenomena, J. Res. Sci. Teach., 34(9), 949–968.
- Kraft A., Strickland A. M. and Bhattacharyya G., (2010), Reasonable reasoning: multi-variate problem-solving in organic chemistry, Chem. Educ. Res. Pract., 11(4), 281–292.
- Krieter F. E. et al., (2016), Thinking like a chemist: Development of a chemistry card-sorting task to probe conceptual expertise, J. Chem. Educ., 93(5), 811–820.
- Lin S.-Y. and Singh C., (2009), Categorization of quantum mechanics problems by professors and students, Eur. J. Phys., 31(1), 57–68.
- Lincoln Y. S. and Guba E. G., (1985), Naturalistic Inquiry, Newbury Park, CA: SAGE Publications, Inc.
- Mason A. and Singh C., (2011), Assessing expertise in introductory physics using categorization task, Physical Review Special Topics – Physics Education Research, 7(2), 1–17.
- Mason A. and Singh C., (2016), Using categorization of problems as an instructional tool to help introductory students learn physics, Phys. Educ., 51(2), 025009.
- Mayer R. E., (2012), Information processing, APA educational psychology handbook, Theories, constructs, and critical issues, Washington: American Psychological Association, vol. 1, pp. 85–99.
- Novak J. D., (1993), Human constructivism: A unification of psychological and epistemological phenomena in meaning making, International Journal of Personal Construct Psychology, 6(2), 167–193.
- Novak J. D., (2002), Meaningful learning: The essential factor for conceptual change in limited or inappropriate propositional hierarchies leading to empowerment of learners, Sci. Educ., 86(4), 548–571.
- Novak J. D., (2010), Learning, creating, and using knowledge, New York, NY: Taylor & Francis Group.
- Rottman B. M., Gentner D. and Goldwater M. B., (2012), Causal systems categories: Differences in novice and expert categorization of causal phenomena, Cognit. Sci., 36(5), 919–932.
- Saldaña J., (2013), The coding manual for qualitative researchers, Thousand Oaks, CA: SAGE Publications, Inc.
- Schunk D. H., (2012), Learning theories: An educational perspective, 6th edn, Los Angeles, CA: SAGE Publications.
- Smith J. I. et al., (2013), Development of the biology card sorting task to measure conceptual expertise in biology, CBE Life Sci. Educ., 12(4), 628–644.
- Smith M. U., (1992), Expertise and the organization of knowledge: Unexpected differences among genetic counselors, faculty, and students on problem categorization tasks, J. Res. Sci. Teach., 29(2), 179–205.
- Snyder J. L., (2000), An investigation of the knowledge structures of experts, intermediates and novices in physics, Int. J. Sci. Educ., 22, 979–992.
- Stains M. and Talanquer V., (2008), Classification of chemical reactions: Stages of expertise, J. Res. Sci. Teach., 45(45), 771–793.
- Sweller J., (1994), Cognitive Load Theory, Learning Difficulty, and Instructional Design, Learn. Instr., 4(4), 295–312.
- Sweller J. and Chandler P., (1994), Why some material is difficult to learn, Cognit. Instr., 12(3), 185–233.
- Taber K. S., (2009), Learning at the Symbolic Level, in Gilbert J. K. and Treagust D. F. (ed.), Multiple Representations in Chemical Education, Dordrecht: Springer Netherlands, pp. 75–104.
- Wolf S. F., Dougherty D. P. and Kortemeyer G., (2012a), Empirical approach to interpreting card-sorting data, Physical Review Special Topics – Physics Education Research, 8(1), 1–15.
- Wolf S. F., Dougherty D. P. and Kortemeyer G., (2012b), Rigging the deck: Selecting good problems for expert-novice card-sorting experiments, Physical Review Special Topics – Physics Education Research, 8(2), 1–7.
- Yamauchi T., (2005), Labeling bias and categorical induction: generative aspects of category information, Journal of experimental psychology. Learning, memory, and cognition, 31(3), 538–553.
| This journal is © The Royal Society of Chemistry 2019 |