View PDF VersionPrevious ArticleNext Article
 
DOI: 10.1039/C6RP00132G (Paper) Chem. Educ. Res. Pract., 2016, 17, 1156-1173

Evaluating students' learning gains and experiences from using http://nomenclature101.com

Nicholas E. Bodé a, Jeanette Caron b and Alison B. Flynn *a
aDepartment of Chemistry and Biomolecular Sciences, University of Ottawa, 10 Marie Curie Private, Ottawa, ON K1N 6N5, Canada. E-mail: alison.flynn@uottawa.ca
bCentre for e-Learning, Teaching and Learning Support Service, University of Ottawa, 136 Jean-Jacques Lussier, Ottawa, ON K1N 6N5, Canada

Received 7th June 2016 , Accepted 2nd September 2016

First published on 2nd September 2016

Abstract

Skill in organic chemistry nomenclature is fundamental for communicating more complex concepts. Interpreting and using functional group names is particularly important. With http://nomenclature101.com, students can create tailored interactive quizzes according to their learning needs; the tool is free and available in English and French. The present study investigated students' nomenclature learning in three different learning environments to determine (1) what learning gains participants make when using http://nomenclature101.com independently, when guided, or taught nomenclature through a traditional classroom tutorial and (2) students' perceptions of the usefulness, ease of use, and overall learning experience in each of the settings. We invited students from first and second semester organic chemistry courses at a large, research-intensive Canadian university to participate in a nomenclature workshop. When participants arrived, they were randomly sorted into one of three treatment groups: classroom tutorial, independent use of nomenclature101, or guided use of nomenclature101. Before the session, participants completed a pre-test; at the end, they completed a post-test and a questionnaire related to affective aspects of their experience. We analyzed participants' scores and questionnaire responses, and qualitatively analyzed their test answers, including errors. Learning gains were significant with large effect sizes for all three groups although there were no significant differences in learning gains between groups. The largest gains were observed in the ability to correctly identify and draw functional groups. Exploratory factor analysis showed that the post-workshop questionnaire about http://nomenclature101.com reliably measured participants’ perceptions about two latent factors: usefulness and ease of use. Based on questionnaire results, most participants liked the learning tool and found it useful and easy to use. Participants in all groups reported enjoying their learning experience. Few participants postdicted their quiz grades accurately, suggesting that metacognitive skillfulness was lacking among workshop participants. The large learning gains observed after using http://nomenclature101.com or learning in a classroom setting for just one hour shows the potential that instruction has to help students learn functional group identification skills, which ideally will mitigate barriers to communication and understanding. These results offer flexibility to educators as they make instructional choices such as teaching nomenclature in a course period or tutorial setting or asking students to learn nomenclature independently with http://nomenclature101.com. Students have the flexibility to work outside of class in the manner of their choosing.

Introduction

Nomenclature skill provides a foundation for learning advanced chemistry. Students must first be able to identify functional groups and name and draw molecules before they can solve advanced problems. While students do not necessarily need to know the correct placement of every comma and dash in a molecule's name, barriers to communication can arise in discussions about reactivity when participants cannot at least identify the functional groups in question. Students in a first year organic chemistry course struggled with these skills, even in the final exam; the class average on a question asking students to identify functional groups (an aldehyde and an alcohol) was only 58%, instead of the expected 98% (Flynn et al., 2013). The results were even lower when students were asked to name a specific molecule. Despite a large number of resources available describing nomenclature, few interactive resources exist that allow students to actively practice and get feedback; even fewer resources exist in French, one of the languages of instruction at the authors' institution. Many resources are instructor controlled (e.g., homework assignments) and charge a subscription fee.

http://Nomenclature101.com is an online learning tool that was designed and created to address these issues (Fig. 1) (Flynn et al., 2014). The tool is designed to help students achieve three learning outcomes associated with nomenclature:


image file: c6rp00132g-f1.tif
Fig. 1 http://Nomenclature101.com's home page, which supports student learning in English or French (Flynn et al., 2014).

(1) Identify a functional group, given the structure

(2) Name a molecule according to IUPAC rules, given the structure

(3) Draw a molecule, given its IUPAC name

Using http://nomenclature101.com, students can create tailored, interactive quizzes according to their learning choices; the tool is free and available in English and French. Students receive feedback (right/wrong) when they submit their answer, can consult hints, and see facts about molecules.

The tool has been accessed more than 10[thin space (1/6-em)]000 times from almost every country in the world. However, students' learning gains and experiences using the learning tool have not been investigated. Studies of interactive learning tools suggest that level of guidance can have a strong effect on students' use of the tool or activity (Bonawitz et al., 2011; Akaygun and Jones, 2014; Chamberlain et al., 2014) and that the level of guidance can benefit students at different times (González-Cruz et al., 2003). The present study measured the learning gains in organic chemistry nomenclature of students using http://nomenclature101.com compared to those learning in a technology-free classroom setting and assessed students' satisfaction with the resources offered by the learning tool.

Guskey evaluation framework

http://nomenclature101.com was designed with a constructivist approach modeled on the strategies of cognitive apprenticeship (Oriol et al., 2010) and aligned with many aspects of meaningful learning (Bretz, 2001; Novak, 2011). This study was designed using part of a Guskey evaluation (Guskey, 2002, 2010), a five-level model that examines participants' (or students'): reactions (level 1), learning (level 2), organizational support and change (level 3), use of new knowledge and skills (level 4), and learning outcomes (level 5).

To evaluate the benefits to student learning of http://nomenclature101.com compared to a traditional nomenclature tutorial, we asked the following research questions:

(1) What learning gains have participants who have used http://nomenclature101.com (independently and when guided) made compared to participants who are taught nomenclature through a traditional classroom tutorial? (Guskey level 5)

(2) What are participants' perceptions of the usefulness, ease of use, and overall learning experience in each learning setting? (Guskey level 1)

Methods

Setting and course

Participants in the study were taking Organic Chemistry I and II courses at a large, research-intensive Canadian university. These courses have been described in previous work (Bodé and Flynn, 2016). Organic Chemistry I is offered in the winter semester of students’ first year of studies, and Organic Chemistry II is offered in the summer and fall semesters. Both of these courses may be taken in either English or French, and consist of two weekly lectures (1.5 hours each, mandatory, lecture or flipped format), and an optional tutorial session (1.5 hours, also called a recitation or discussion group). The Organic I course has a required, associated laboratory section (3 hours biweekly) and the Organic II course has a laboratory course that runs concurrently and is only required for some programs (3 hours weekly). Assessment for the course generally consists of two midterm exams, a final exam, online homework assignments, and class participation using a classroom response system (“Top Hat,” 2016). Assessment for the laboratory portion of Organic Chemistry I consists of pre-lab quizzes, completed online before each experiment is performed, six lab reports (one for each experiment), and in-lab performance. These courses are composed of ∼75% Faculty of Science students, ∼17% Faculty of Health Sciences students, and ∼8% students from other faculties. A new curriculum has been used in the courses since 2012 (Flynn and Ogilvie, 2015). The topics taught in Organic Chemistry I and II include: structure, properties, stereochemistry, and conformational analysis of organic compounds, IR and 1H NMR spectroscopy, electron-pushing formalism/symbolism, acid–base chemistry, π bond electrophiles (e.g., 1,2-carbonyl addition reactions, acetals and imine formation, addition–elimination reactions, etc.), π bond nucleophiles, aromaticity and electrophilic aromatic substitution, SN1, SN2, E1, E2, and α-carbons as nucleophiles (e.g., aldol condensation, alkylation).

Participants

The participant pool was composed of 65% Faculty of Science students (N = 45), ∼20% Faculty of Health Science students (N = 13), and ∼12% Faculty of Engineering students (N = 8); the remainder did not report their program of study (N = 3). 52% of participants (N = 36) were first year students, 33% (N = 23) were second year students, 6% (N = 4) were third year students, and 4% (N = 3) were fourth year students; the remaining 4% (N = 3) did not report their year of study. Students from organic chemistry courses were verbally invited to participate in a nomenclature workshop, which was held outside of regular course hours. The authors' institutional review board (IRB) reviewed and approved all stages of the research project. Informed consent was obtained for all stages involving human subjects.

The workshop was piloted in the fall of 2014, then conducted at two different times: February 2015 in English and French and May 2015 in English. In February 2015, participants were recruited from Organic Chemistry I courses; in May 2015, participants were recruited from Organic Chemistry II. 59% of participants (N = 41) completed the workshop in February 2015, while the remaining 41% of participants (N = 28) completed the workshop in May 2015. 85% of participants completed the workshop in English (N = 59), and 15% completed the workshop in French (N = 10).

Session formats

When participants arrived for the workshop, they were randomly assigned to one of three, one-hour tutorial sessions on organic chemistry nomenclature (Fig. 2). Participants who did not bring a laptop or mobile device (as requested) used the computer in the room or worked with another participant.
image file: c6rp00132g-f2.tif
Fig. 2 Methods for the study. The first workshop (February 2015) was offered in both English and French; the second workshop (May 2015) was only offered in English.

All the sessions were guided by videos to ensure consistency between sections (French/English and February/May). All videos welcomed the participants, described the purpose of the study (but did not tell participants which group they were in), and asked participants to complete the pre-test while the video paused. After the pre-test, the videos described the intended learning outcomes (ILOs) for the session (same for each group) with a screenshot of http://nomenclature101.com (without the site title or URL present) (Fig. 3); the ILOs were also handed out to the participants. Then Group 1 received a traditional classroom tutorial on organic nomenclature, Group 2 used http://nomenclature101.com independently, and Group 3 received a guided tutorial on http://nomenclature101.com (Fig. 2) (note: we capitalized “Group” when naming one of these three groups in our results and discussion for clarity). At the end of the session, the video was restarted, participants were instructed to complete the post-test and questionnaire, and they were thanked for attending. These portions of the videos were identical for all three groups. The workshops took approximately two hours to complete, with 45–60 minutes for the tutorial session itself.


image file: c6rp00132g-f3.tif
Fig. 3 Intended learning outcomes that were given to all workshop participants via video and handout.

The classroom sessions in Group 1 were facilitated by PhD candidates in organic chemistry with multiple years of experience as teaching assistants for organic chemistry courses. After the video introduction, these facilitators presented a one-hour lecture on nomenclature punctuated with exercises for the participants to complete; they answered participants' questions as they arose. During the exercises, participants could work individually or in groups.

After the video introduction, Group 2 (unguided use of http://nomenclature101.com) participants were prompted to use http://nomenclature101.com for one hour. They could work individually or in groups. Facilitators for Group 2 could answer technical questions about the site but not chemistry questions.

After the video introduction, Group 3 (guided use of http://nomenclature101.com) participants were prompted to explore http://nomenclature101.com for 10 minutes in any way they wanted. Next, Group 3 received additional video guidance that Groups 1 and 2 did not receive, which explained the various features of the learning tool, including the resources available (e.g., find hints, see nomenclature rules, see facts about the molecules, use the quiz dashboard, see a summary of results). Participants could work individually or in groups. The next 5 minutes of video gave recommendations on how to work through the site: first, to practice identifying functional groups then to drawing molecules (going from the simplest to the most complex and those with stereocentres). Participants were also advised to work on aspects of the site that were aligned with the session's ILOs. Participants then had 45 minutes to use the site, for a total of one hour learning time. They could work individually or in groups. The facilitators for Group 3 were chemistry professors or senior teaching assistants who could answer both technical and chemistry questions.

At the beginning of the session, all groups completed the same pre-test on organic chemistry nomenclature (Appendix 1). The pre-test had twelve questions: six questions asked participants to identify functional groups (given a complex structure) and six questions asked participants to draw molecules (given the name). Immediately following the workshops, participants completed a post-test on organic chemistry nomenclature (Appendix 2). This test was in the same format as the first, with different but aligned questions; all groups completed the same post-test. The pre-test and post-test contained the same questions in the English/French and February/May sessions. These questions were aligned with the expected nomenclature learning outcomes from the Organic I course.

Lastly, participants completed a questionnaire (Appendices 3 and 4). The questionnaire was adapted from an existing one (Bolliger and Supanakorn, 2011) and was designed to access students' perception of the tutorial based on four dimensions—usefulness, access, interface and learner performance. Each questionnaire contained sixteen questions answered using a Likert scale from 1 (strongly disagree)—5 (strongly agree). Twelve items were positively worded and four were negatively worded. The items were worded in the same way on each questionnaire, but modified to address the setting; for example, tutorial setting (Group 1) Item 1 statement, “The tutorial was useful in learning nomenclature”, was reworded: “The quizzes available in http://www.nomenclature101.com were useful in learning nomenclature” on the http://nomenclature101.com workshop questionnaire (Groups 2 and 3). The questionnaire also asked participants to report demographic information and their expected letter grade in the workshop.

Data analysis

We analyzed participants' scores, types and frequencies of errors, and questionnaire responses, making comparisons between the pre- and post-tests, between groups, and within groups. We performed all statistical analyses using IBM SPSS Statistics (version 23). We combined the results from groups that completed the workshop in February and in May because there were no significant differences between their test scores on either the pre-test, U = 502.0, z = −0.391, p = 0.696, or the post-test, U = 508.5, z = −0.306, p = 0.760. The students who participated in the workshop in May had just started Organic Chemistry II, and had already completed Organic Chemistry I, while those who participated in February had not completed a full semester of organic. As such, we might expect that test scores would be higher in the May workshop, as these participants would likely have slightly more prior chemistry knowledge. A possible reason that there was no significant difference in scores is that students who take Organic Chemistry II in the summer may be weaker students who were repeating the course, because students following the Faculty's suggested course sequence normally take this course in the Fall semester. Another possibility is that there was little explicit nomenclature instruction in Organic Chemistry I (nomenclature learning outcomes were communicated to students in the syllabus and they were expected to work independently), and participants in May had not yet been tested on nomenclature in their Organic Chemistry II course.

We also combined the results from groups that completed the workshop in French with those that completed the workshop in English, because there were no significant differences between their test scores on either the pre-test, U = 257.5, z = −0.404, p = 0.686, or the post-test, U = 256.5, z = −0.422, p = 0.673.

To determine the effect of the nomenclature tutorial and of using http://nomenclature101.com (independently and when guided) on participants' test scores, we performed two-way repeated measures ANOVA. The dependent variable in each ANOVA was participants' scores on the tests that were administered. The within-subjects variable was time (pre-test and post-test) and the between-subjects factor was the workshop treatment group. The main effect of time and the interaction effects of time and the workshop treatment group were evaluated using the criterion of Wilks' lambda (Λ). We performed a parametric ANOVA instead of a non-parametric test, despite having non-normal data. The parametric test was appropriate because of the larger overall sample size (N = 66) and an observed power of 1, which indicated a low probability of committing type II errors.

For follow-up analysis of the time effect and due to the smaller sample sizes for the individual workshop groups, we performed Kolmogorov–Smirnov tests of normality on the data to determine the suitability of using t-tests. The distribution of scores was significantly non-normal (p < 0.05) for all groups on the pre-test, and for Groups 1 and 2 on the post-test. Therefore, we conducted Mann–Whitney U tests, the non-parametric equivalent of the t-test, to perform post hoc analysis of the significant main effect of time in a more statistically powerful fashion. We report effect size measures for Mann–Whitney U tests using Pearson's r.

To analyze participants' errors on the “circle the functional group” questions, we identified the functional group that each participant circled. To analyze the “draw the molecule” questions, one author coded participants' responses to each question type for certain types of errors using an open-coding system (Patton, 2014); the codes are summarized in Table 1. A second rater then independently coded a subset of the data using the same coding template to determine the degree of reliability in our interpretation of participants' errors in the “draw the molecule” questions. We measured the reliability of the coding system using the criterion of Krippendorff's α because this statistic adjusts for chance agreement, while a simple percent agreement calculation does not (Harshman and Yezierski, 2015). The results of this analysis show that our coding system was reliably applied throughout the data (6 codes, Nitems = 403, α = 0.791). Krippendorff suggests that a value above 0.800 is ideal, but any value above 0.667 is still acceptable to draw any conclusions from a content analysis (Krippendorff, 2004). Once our reliability analysis was complete, the data were recoded until both raters reached complete agreement upon all codes.

Table 1 Coding system used to analyze “draw the molecule” questions. Reliability analysis was performed using the criterion of Krippendorff's α
Code Description
a Only relevant to Question 11.
Missing/extra carbon Number of carbon atoms in the molecule does not match the name (either in the backbone or branches)
Incorrect constitutional isomer Functional group(s) are not bound to the carbon atom indicated in the name of the molecule
Incorrect or missing stereochemistrya Configuration at stereocentre is not indicated using correct symbolic conventions or incorrect configuration is indicated
Incorrect or missing functional group Functional group drawn does not match the name, or a functional group indicated in the name is missing from the molecule drawn
Non-meaningful use of symbolism Parts of a molecule are drawn (e.g., functional groups, Lewis structures) that do not follow the conventions for accurately representing organic molecules

To assess the significance of differences in error frequencies between groups and between pre-workshop and post-workshop tests, we performed chi-square (χ2) tests of independence. We report effect size measures for chi-square tests using the criteria of Cramér's V (ϕc).

Using participants' responses to the post-workshop http://nomenclature101.com questionnaire, we performed principal axis factoring with oblique rotation (Promax with Kaiser normalization). For the questionnaire responses, we reverse-coded negatively worded items (item no. 5, 9, 12, 14—Table 3) to facilitate analysis. We performed reliability analysis to verify the internal consistency of both the http://nomenclature101.com and nomenclature tutorial questionnaires using Cronbach's α. We also used the questionnaires to assess participants' postdiction accuracy by asking them what they thought their letter grade on the post-test would be. Participants' responses to this item were compared to their actual scores to gain insight about participants' metacognitive skillfulness.

Limitations

Social constructivist theories of learning propose that limitations on an individual's level of potential development are determined through collaborative rather than individual problem solving (Vygotsky, 1978); as such, a factor that may have influenced learning gains was whether or not participants chose to spend their time using http://nomenclature101.com alone or in groups. We hypothesize that participants who chose to work in groups would have benefitted more from the workshop than those who chose to work alone, but social effects on learning were not measured as part of the study.

We also did not measure the extent to which learning gains from the workshop were retained on a long-term basis, but expect that regular, long-term use of http://nomenclature101.com would be required to ensure that any learning gains would be retained (Kornell, 2009). Future studies that measure learning gains following an intervention could consider the retention aspect of learning, but controlling for variations in what participants do between a post-test and a delayed post-test (for instance, how much/how often they choose to review the material) may be a challenge.

Results and discussion

All groups showed large learning gains

Our first goal was to determine the learning gains that participants experienced using this tool compared to a traditional tutorial setting. The majority of participants in all groups performed better on the post-workshop test than on the pre-workshop test (Fig. 4 and 5).
image file: c6rp00132g-f4.tif
Fig. 4 Box plots showing the distribution of pre-test and post-test scores for each treatment group. Ntutorial = 20, Nhttp://nomenclature101.com[thin space (1/6-em)](no[thin space (1/6-em)]guidance) = 21, Nhttp://nomenclature101.com[thin space (1/6-em)](with[thin space (1/6-em)]guidance) = 25.

image file: c6rp00132g-f5.tif
Fig. 5 Most participants' post-test scores were higher than their pre-test scores. Ntutorial = 20, Nhttp://nomenclature101.com[thin space (1/6-em)](no[thin space (1/6-em)]guidance) = 21, Nhttp://nomenclature101.com[thin space (1/6-em)](with[thin space (1/6-em)]guidance) = 25.

Learning gains were significant for all groups and there was no significant difference in learning gains between groups (Table 2). The results of a two-way repeated measures ANOVA show the main effect of time was significant, Λ = 0.498, F(1,63) = 63.480, p < 0.001, and had a large effect size, partial η2 = 0.50. The interaction effect was non-significant, Λ = 0.990, F(2,63) = 0.327, p = 0.722, with a small effect size, partial η2 = 0.01. The test of between-subjects effects shows that the effect of the workshop treatment was non-significant, F(2) = 0.01, p = 0.990, partial η2 < 0.001. Post hoc analysis showed the learning gains in each individual workshop were significant and had medium effect sizes (Table 2).

Table 2 Mann–Whitney U test results, comparing nomenclature pre-test and post-test total scores
Group N pre-test U z p r
a N pre-test = Npost-test. b Medium effect size.
Tutorial (Group 1) 20 100.0 −2.712 0.007 −0.429b
Unguided use of http://nomenclature101.com (Group 2) 21 101.5 −3.017 0.003 −0.466b
Guided use of http://nomenclature101.com (Group 3) 25 167.0 −2.829 0.006 −0.400b

Functional group identification: errors and learning gains. All groups had significant gains in the ability to identify functional groups and there were no significant differences between groups. Two-way repeated measures ANOVA showed the time main effect was significant, Λ = 0.637, F(1,63) = 35.953, p < 0.001, with a large effect size, partial η2 = 0.36. The interaction effect was non-significant, Λ = 0.949, F(2,63) = 1.683, p = 0.19, with a small effect size, partial η2 = 0.05. The test of between-subjects effects shows that the effect of the workshop treatment was non-significant, F(2) = 0.025, p = 0.976, partial η2 = 0.001.

The most pronounced gains related to functional group identification were in the ability to identify an acetal; participants commonly identified an ester on the pre-test when asked to identify an acetal, but many corrected this error on the post-test (Fig. 6). The increase in the ability to identify this functional group was significant with a large effect size for Group 1, χ2(1) = 18.027, p < 0.001, ϕc = 0.67, significant with a medium effect size for Group 2, χ2(1) = 9.882, p = 0.002, ϕc = 0.49, and significant with a large effect size for Group 3, χ2(1) = 14.286, p < 0.001, ϕc = 0.54.


image file: c6rp00132g-f6.tif
Fig. 6 Types of functional groups circled by workshop participants on (A) pre-test question 3 and (B) post-test question 3. The correct answer to each question is indicated with a box on the molecule and a checkmark on the chart. Ntutorial = 20, Nhttp://nomenclature101.com[thin space (1/6-em)](no[thin space (1/6-em)]guidance) = 21, Nhttp://nomenclature101.com[thin space (1/6-em)](with[thin space (1/6-em)]guidance) = 25. Participants made significant gains in functional group identification between the pre- and post-tests in every workshop group.

Close to significant gains were observed for Group 1 in the ability to distinguish between a nitro group and a nitrile, χ2(1) = 3.636, p = 0.057, with a medium effect size, ϕc = 0.30. Non-significant gains were observed for Group 2 participants, χ2(1) = 1.556, p = 0.212, with a small effect size, ϕc = 0.19, and for Group 3 participants, χ2(1) = 1.389, p = 0.239, with a small effect size, ϕc = 0.17 (Fig. 7). Although learning gains were not significant for any Group, more than 65% of participants in each group correctly identified a nitrile on the post-test.


image file: c6rp00132g-f7.tif
Fig. 7 Types of functional groups circled by workshop participants on (C) pre-test question 4 and (D) post-test question 4. The correct answer to each question is indicated with a box on the molecule and a checkmark on the chart. Ntutorial = 20, Nhttp://nomenclature101.com[thin space (1/6-em)](no[thin space (1/6-em)]guidance) = 21, Nhttp://nomenclature101.com[thin space (1/6-em)](with[thin space (1/6-em)]guidance) = 25.

The difference between an ester and an acetal, which have a similar arrangement of atoms, and between a nitro group and a nitrile, which have similar sounding names, might be considered trivial to the novice, but represent important implications in structure and reactivity. The fact that these learning gains can be observed after just one hour using http://nomenclature101.com shows the potential this learning tool has to mitigate the small errors that could lead to much larger errors in understanding reactivity later on in the course, without an impact on classroom time. Alternatively, one hour class time dedicated to nomenclature could rapidly lead to learning gains.

Drawing molecules given their name: errors and learning gains. All groups made significant gains in the ability to draw molecules although there was no significant difference between groups in gains in the ability to draw molecules. Two-way repeated measures ANOVA showed the main effect of time was significant for this section of the test, Λ = 0.717, F(1, 63) = 24.815, p < 0.001, with a large effect size, partial η2 = 0.28. The interaction effect was non-significant, Λ = 0.970, F(2,63) = 0.986, p = 0.379, with a small effect size, partial η2 = 0.03. The test of between-subjects effects shows that the effect of the workshop treatment was non-significant, F(2) = 0.013, p = 0.987, partial η2 < 0.001.

Fig. 8 shows the proportions of participants who made each type of error on at least one of the six questions, and shows that each of those frequencies decreased for http://nomenclature101.com users between the pre-test and post-test. The most pronounced decrease among http://nomenclature101.com users (Groups 2 and 3) was in incorrect or missing functional groups. This improvement was significant for Group 2, χ2(1) = 6.857, p = 0.009, with a medium effect size, ϕc = 0.40, and significant for Group 3, χ2(1) = 5.903, p = 0.015, with a medium effect size, ϕc = 0.36. Meanwhile, the decrease was non-significant in tutorial participants who made incorrect or missing functional group errors, χ2(1) = 1.600, p = 0.206, with a small effect size, ϕc = 0.20. The organization of the test bank, which is sorted by functional group, is a possible reason participants using http://nomenclature101.com may have experienced more significant gains in this area.


image file: c6rp00132g-f8.tif
Fig. 8 Types of errors made by participants on the “draw the molecule” questions on pre-workshop and post-workshop tests. Frequencies indicate the proportion of participants who made the indicated error on at least one of the six questions. Ntutorial = 20, Nhttp://nomenclature101.com[thin space (1/6-em)](Unguided) = 21, Nhttp://nomenclature101.com[thin space (1/6-em)](Unguided) = 25.

One area where tutorial participants appeared to improve much more than http://nomenclature101.com users was in the ability to draw the correct configuration at stereocentres (Fig. 8). The decrease in this error was significant for tutorial participants (Group 1), χ2(1) = 4.912, p = 0.027, with a medium effect size, ϕc = 0.35. However, there was no significant change in this error for Group 2, χ2(1) = 1.615, p = 0.341, with a small effect size, ϕc = 0.20, or Group 3 participants, χ2(1) = 0.347, p = 0.769, with a small effect size, ϕc = 0.08. A possible reason for this difference is that http://nomenclature101.com users were free to practice whichever question types they wanted and may not have chosen to focus on stereochemistry. However, this may illustrate a lack in metacognitive skillfulness among the participants if they are unable to identify these gaps in their understanding of organic nomenclature and practice accordingly.

We found the non-meaningful use of symbolism (Fig. 9) in nearly one-third of participants' answers, but less commonly among http://nomenclature101.com users than among tutorial participants. While there was no change in frequency of this error among tutorial participants, there was a slight (but non-significant) decrease in the frequency of this error among Group 2 participants, and the decrease in the frequency of this error among participants in Group 3 was close to significant, χ2(1) = 3.388, p = 0.066, with a small effect size, ϕc = 0.26. This may have been a less common error among http://nomenclature101.com users because the drawing tool does not allow users to draw functional groups in the non-meaningful fashion; furthermore, the learning tool shows the answers with the molecules correctly drawn.


image file: c6rp00132g-f9.tif
Fig. 9 A common error was the non-meaningful use of symbolism when drawing functional groups (right).

Questionnaire analysis: factor analysis and reliability

To answer our second research question, we first investigated whether the http://nomenclature101.com questionnaire reliably measured participants' perceptions of the usefulness and ease of use of the website (Table 3). The questionnaires are given in Appendices 3 and 4. We performed exploratory factor analysis using participants' responses to see if the questionnaire could reliably measure these latent factors (usefulness and ease of use). The Kaiser–Meyer–Olkin measure verified that our sample size was adequate, KMO = 0.852, which is well above the acceptable limit of 0.500 (Kaiser, 1974).
Table 3 http://nomenclature101.com questionnaire (Groups 2 and 3): principal axis factoring and reliability analysis results
Subscale mean Item no. Item statements, by subscales Rotated factor loadingsa Mean Std. dev. α if item deleted
1 2
Perceived usefulness (N = 42)
a Values less than 0.3 are omitted. b Items are reverse-coded (i.e., strongly disagree = 5, strongly agree = 1).
3.95 1 The quizzes available in http://www.nomenclature101.com were useful in learning nomenclature. 0.804 4.10 1.10 0.920
2 I enjoyed using the nomenclature quizzes as a way to review the material that I learned in class. 0.922 3.83 1.17 0.918
3 The nomenclature quizzes aided me in understanding the material. 0.793 3.93 1.13 0.924
4 I will do other nomenclature quizzes in the future. 0.768 4.17 0.93 0.926
5b It is faster to study the material myself than to learn from the nomenclature quizzes. 0.317 3.33b 1.00 0.948
6 The http://www.nomenclature101.com website helped me improve my skill. 1.026 3.98 0.92 0.920
7 I believe using http://www.nomenclature101.com will help improve my course grade. 0.905 3.98 1.00 0.922
8 I appreciate that I will be able to access http://www.nomenclature101.com anywhere (with WiFi). 0.796 4.31 1.02 0.928
Perceived Ease of Use (N = 45)
3.94 9b Technical difficulties using http://www.nomenclature101.com obstructed my experience. 0.562 3.67b 1.40 0.895
10 I like that I am able to use http://www.nomenclature101.com outside of class. 0.493 0.443 4.33 0.85 0.881
11 I felt comfortable using http://www.nomenclature101.com. 0.669 4.18 1.07 0.875
12b I had a hard time navigating http://www.nomenclature101.com. 0.717 3.93b 1.14 0.882
13 The http://www.nomenclature101.com is well designed. 0.798 3.93 1.14 0.876
14b The pace of the activity on http://www.nomenclature101.com was too fast. 0.524 3.93b 0.78 0.908
15 The length of time allocated to ‘study’ using http://www.nomenclature101.com was appropriate. 0.796 3.53 1.12 0.888
16 The consistent interface design made it easy for me to follow the material. 0.806 4.02 0.89 0.882
Observed eigenvalues 9.202 1.515
Estimated 95% confidence interval of the eigenvalues [5.994,13.00] [0.889,2.141]
% of variance 57.51 9.468
α 0.935 0.899

We performed an initial factor analysis following Kaiser's criterion, retaining factors with an eigenvalue greater than 1. This analysis yielded 3 factors that, in combination, explained 74.65% of the variance. We then estimated 95% confidence intervals for each eigenvalue, following the revised Kaiser's criterion to retain factors with eigenvalues where the entire 95% confidence interval was greater than 1. This procedure helps avoid retaining too many factors due to errors related to sample size (Larsen and Warne, 2010).

We chose to retain two factors that, in combination, explained 66.98% of the variance, but a small portion of the estimated 95% CI for our second factor is below 1 (95% CI = [0.889,2.141]). We decided to keep this second factor because of the qualitatively thematic differences between the items that loaded onto the different factors, which are supported by reasonably high factor loadings. Underlying themes between the items loading on factor 1 suggest that this factor represents participants' perceptions of the usefulness of http://nomenclature101.com, while factor 2 represents participants’ perceptions of the ease of use of http://nomenclature101.com. Although questionnaire item 10 loads more strongly onto factor 1, we believe that the statement “I like that I am able to use http://www.nomenclature101.com outside of class” is more relevant to http://nomenclature101.com's ease of use than its usefulness. As each of these factors contains at least four items with loadings of 0.60 or greater, these latent factors revealed by the pattern matrix can be reliably interpreted (Guadagnoli and Velicer, 1988).

Reliability analysis of each latent factor indicated excellent homogeneity within the subscales pertaining to perceived usefulness and perceived ease of use of http://nomenclature101.com (Cronbach's α = 0.935 and 0.899 for factors 1 and 2 respectively). Homogeneity is a measure of the extent to which there is a single latent variable that is common to the items in the domain being measured (Revelle and Zinbarg, 2008). A limitation of this reliability analysis is that the α coefficient is intended to be used as a reliability measure of normally distributed data, while the distribution of responses to these Likert-scale questionnaires is non-normal (as is often the case) (Bretz and McClary, 2015).

We also performed principal axis factoring using participants' responses to the tutorial questionnaire (Group 1), but the obtained KMO measure of 0.452 indicated inadequate sampling, as it is below the suggested threshold for exploratory factor analysis of 0.500 (Kaiser, 1974). As such, the results of factor analysis were disregarded, and the questionnaire as a whole was analyzed for reliability. A Cronbach's α value of 0.806 was obtained, indicating good internal consistency of the questionnaire as a whole. A possible limitation of this statistic is the small sample size (N = 17) from which it was obtained.

Participants thought http://nomenclature101.com was useful and easy to use

The Likert scale scores were fairly high across all statements indicating that participants generally agreed with positively worded items and disagreed with negatively worded items on both questionnaires and that participants in different groups responded to each item in a similar fashion. Boxplots comparing the distribution of scores between groups on individual questionnaire items can be found in Appendix 5. The subscale means that were obtained indicate that participants agree http://nomenclature101.com is both useful (subscale mean = 3.95) and easy to use (subscale mean = 3.94) (Table 3). Specific features that some participants liked about http://nomenclature101.com included the test bank's organization and large size, along with the option to customize the difficulty of quizzes. Participants also had some suggestions for improving the learning tool, such as adding the option to customize the number of quiz questions. The post-tutorial questionnaires were not compared in depth between groups. This is because differences in the learning environments may have resulted in possible differences in how the item statements would be interpreted by participants, and as such, these comparisons would not be meaningful.

Comparably high means on analogous items demonstrate that both learning environments were useful, and therefore, environment-related factors such as the pace of the workshops, the quality of the workshop facilitators, and participant comfort levels were likely not different enough between groups to confound the results of this study.

The Group 1 participant who obtained a score of 0 on both the pre- and post-tests (Fig. 5) provided many comments in their questionnaire, including: “the facilitator was good at answering questions” but he went “too fast, he thought we knew everything already (I've never seen any of this before)” and “I really like how Dr. [name] does videos outside of class. That way you can pause and take good notes instead of trying to rush it.” Although the tutorial was designed to build off of knowledge that we expected most participants to have, this participant did not have the anticipated prior knowledge. A site such as http://nomenclature101.com can benefit that type of student because the student can start learning at any point, as long as the student can identify where they should most appropriately begin.

Participants' self-evaluations were inaccurate

One of the primary factors that can limit a student's potential gains in a self-regulated learning environment is their metacognitive skillfulness, which refers to the ability of the student to assess their knowledge and regulate their cognition (Azevedo et al., 2012). In a learning environment where the student must independently self-regulate their learning, they must constantly monitor their learning and adapt their strategies as necessary in order to be successful. Aspects of metacognitive skillfulness may include determining what strategies to use to solve problems and setting well-defined learning goals (Azevedo et al., 2012).

The process of setting learning goals should also be informed by one's current levels of knowledge and understanding; therefore, another important aspect of metacognitive skillfulness is the ability to accurately self-assess knowledge and understanding. We asked participants to postdict (i.e., “predict” after the fact) their score on an assessment task (Hawker et al., 2016). Some student-centered learning environments (SCLEs) in science education exist that were designed using theories of metacognition and self-regulated learning (Azevedo et al., 2010; Kinnebrew and Biswas, 2011) but http://nomenclature101.com does not explicitly guide or scaffold students' metacognitive thinking and the Group 1 tutorial setting moved at a unified pace based on estimated prior knowledge and processing ability.

We compared participants' self-reports of their expected grade on the post-test (postdictions) with their actual post-test scores. Because http://nomenclature101.com users receive constant feedback using the website, we hypothesized that Group 2 and 3 participants would make more accurate postdictions than the Group 1 (tutorial) participants. We further expected Group 3 participants to make the most accurate postdictions because they were encouraged to work according to a gradient of difficulty and according to their individual existing knowledge and abilities.

However, between groups, there was no significant difference in postdiction accuracy (Fisher's exact test: p = 0.980); weaker participants consistently over-estimated their scores on the post-test (Fig. 10). These trends in postdiction accuracy are consistent with findings from other studies (Karatjas, 2013; Hawker et al., 2016) and with research showing that less skilled individuals often believe they are more competent than they are in reality, a cognitive bias commonly called the Dunning–Kruger effect (Kruger and Dunning, 1999).


image file: c6rp00132g-f10.tif
Fig. 10 Postdiction accuracy of tutorial and http://nomenclature101.com workshop participants. Participants were asked to postdict (i.e., after completing the test) their letter grade on the post-test; this postdiction was compared to their actual score. NGroup[thin space (1/6-em)]1 = 20, NGroup[thin space (1/6-em)]2 = 21, NGroup[thin space (1/6-em)]3 = 25.

We also asked Group 2 and 3 participants how often they would use http://nomenclature101.com and how often they recommended the tool be used. Nine participants who predicted they would get below an A– (<80%) on the post-test recommended that others use the tool more often than they said they would use it themselves. These lower achieving individuals seemed to be aware that they had room for improvement and recommended that other students use http://nomenclature101.com at least once per week; however, most of these individuals said they would not use the website more than once every two weeks. Their choices to use the website less often could be due to other conflicting course or academic priorities or other personal reasons.

Metacognitive skillfulness is essential to optimizing learning gains from a self-directed learning environment such as http://nomenclature101.com. Based on the significant learning gains for participants that used this learning tool, we expected that they would also be reasonably good at predicting their scores, due in part to both high metacognitive skillfulness and the features of the site that facilitate the development of these skills, such as the constant feedback. One possibility is that longer-term use of the tool or further metacognitive guidance or scaffolding are needed to develop these skills.

Conclusions and implications for instruction

This study compared students' learning gains on organic chemistry nomenclature problems after one of three treatments and used a questionnaire to investigate students' perceptions of the usefulness, ease of use, and overall learning experience in their Group's learning setting. Participants were randomly sorted into one of three groups; they completed a pre-test, participated in a one-hour session where they either received a nomenclature tutorial (Group 1), used http://nomenclature101.com without guidance (Group 2), or used http://nomenclature101.com with guidance (Group 3), then completed a post-test and questionnaire.

The learning gains were large across all three groups although there were no significant differences between groups. The most evident learning gains by the two groups that used http://nomenclature101.com were in the ability to correctly name and identify functional groups; this was a common error on pre-test questions that was significantly less frequent on post-test questions. http://nomenclature101.com has the potential to mitigate the small errors that could lead to much larger errors in understanding reactivity later on in the course.

Participants in all groups reported high levels of satisfaction with their experiences and participants in Groups 2 and 3 generally agreed with the items pertaining to the characteristics of the website. However, participants were generally inaccurate in their postdictions of their performance in the workshop, especially students with lower grades. These results are consistent with metacognition research (Kruger and Dunning, 1999; Karatjas, 2013; Hawker et al., 2016) and suggest that participants are either lacking in metacognitive skillfulness (which we did not measure beyond postdiction accuracy) or that http://nomenclature101.com does not effectively foster the metacognitive skillfulness that is required for student-centered learning environments to be as effective as they can be. We recommend that the importance of metacognitive skillfulness be emphasized when this tool is used as part of a course and that students be provided with clear guidelines for setting effective learning goals. For example, we share the intended learning outcomes with students using a screenshot of http://nomenclature101.com's homepage (e.g., Fig. 3) and recommend that they identify objectives they have not yet mastered (e.g., by doing a quiz), then work from the simplest to the most complex tasks.

Our results show that http://nomenclature101.com can be a useful resource for students to practice their skills pertaining to organic nomenclature and seems to be an effective substitute for class time, if instructors wish to use class time in other ways. The most pronounced area of weakness for many Group 2 and 3 participants was the ability to draw the correct configuration of functional groups at stereocentres. As such, we recommend that class or tutorial time still be dedicated to teaching this skill or that learning outcomes related to stereochemical nomenclature be made more explicit, so that students can reflect on their skillfulness in identifying/producing the correct configuration at stereocentres more effectively.

Future work

Future research could investigate how the incorporation of this resource into the organic chemistry curriculum is advocated and supported by faculty. Future work could also measure learning gains from the regular use of http://nomenclature101.com in a more longitudinal fashion. Metacognitive skillfulness of the research participants could be studied in more depth, for example by using existing instruments (Cooper and Sandi Urena, 2009) to measure student-reported gains in metacognitive skillfulness associated with a similar intervention as reported in this study. Metacognitive scaffolding could also be built into future online learning tools. Finally, the impact of greater (or lesser) mastery of organic chemistry nomenclature on students understanding of chemistry concepts could be studied in more depth.

Appendix 1. Pre-test


image file: c6rp00132g-u1.tif

image file: c6rp00132g-u2.tif

Appendix 2. Post-test


image file: c6rp00132g-u3.tif

image file: c6rp00132g-u4.tif

Appendix 3. Questionnaire for Group 1


image file: c6rp00132g-u5.tif

image file: c6rp00132g-u6.tif

Appendix 4. Questionnaire for Groups 2 and 3


image file: c6rp00132g-u7.tif

image file: c6rp00132g-u8.tif

Appendix 5. Distribution of responses to questionnaire items


image file: c6rp00132g-u9.tif

Acknowledgements

We thank the University of Ottawa for funding this research, Hayley Sprigings for her contributions, Bradley Cousins and Samuel Pazicni for helpful discussions, facilitators for giving their time and expertise, and the participants.

References

  1. Akaygun S. and Jones L. L., (2014), How Does Level of Guidance Affect Understanding When Students Use a Dynamic Simulation of Liquid–Vapor Equilibrium? in Devetak I. and Glažar S. A. (ed.) Learning with Understanding in the Chemistry Classroom, Dordrecht: Springer Netherlands, pp. 243–263. DOI: http://10.1007/978-94-007-4366-3_13.
  2. Azevedo R., Johnson A., Chauncey A. and Burkett C., (2010), Self-regulated Learning with MetaTutor: Advancing the Science of Learning with MetaCognitive Tools, in Khine M. S. and Saleh I. M. (ed.) New Science of Learning, New York, NY: Springer New York, pp. 225–247, DOI: http://10.1007/978-1-4419-5716-0_11.
  3. Azevedo R., Behnagh R. F., Duffy M., Harley J. M. and Trevors G., (2012), Metacognition and Self-Regulated Learning in Student-Centered Learning Environments, in D. Jonassen and S. Land (ed.) Theoretical Foundations of Learning Environments, 2nd edn, New York, NY: Prentice Hall.
  4. Bodé N. E. and Flynn A. B., (2016), Strategies of Successful Synthesis Solutions: Mapping, Mechanisms and More, J. Chem. Educ., 93(4), 593–604. DOI: http://10.1021/acs.jchemed.5b00900.
  5. Bolliger D. U. and Supanakorn S., (2011), Learning styles and student perceptions of the use of interactive online tutorials, Br. J. Educ. Technol., 42(3), 470–481. DOI: http://10.1111/j.1467-8535.2009.01037.x.
  6. Bonawitz E., Shafto P., Gweon H., Goodman N. D., Spelke E. and Schulz L., (2011), The double-edged sword of pedagogy: instruction limits spontaneous exploration and discovery, Cognition, 120(3), 322–330. DOI: http://10.1016/j.cognition.2010.10.001.
  7. Bretz S. L., (2001), Novak's Theory of Education: Human Constructivism and Meaningful Learning, J. Chem. Educ., 78(8), 1107, DOI: http://10.1021/ed078p1107.6.
  8. Bretz S. L. and McClary L., (2015), Students' Understandings of Acid Strength: How Meaningful Is Reliability When Measuring Alternative Conceptions? J. Chem. Educ., 92(2), 212–219, DOI: http://10.1021/ed5005195.
  9. Chamberlain J. M., Lancaster K., Parson R. and Perkins K. K., (2014), How guidance affects student engagement with an interactive simulation, Chem. Educ. Res. Pract., 15(4), 628–638, DOI: 10.1039/C4RP00009A.
  10. Cooper M. M. and Sandi Urena S., (2009), Design and Validation of an Instrument To Assess Metacognitive Skillfulness in Chemistry Problem Solving, J. Chem. Educ., 86(2), 240–246, DOI: http://10.1021/ed086p240.
  11. Flynn A. B. and Ogilvie W. W., (2015), Mechanisms before Reactions: A Mechanistic Approach to the Organic Chemistry Curriculum Based on Patterns of Electron Flow, J. Chem. Educ., 92(5), 803–810, DOI: http://10.1021/ed500284d.
  12. Flynn A. B., Caron J., Laroche J., Marcoux C. and Richard G., (2013), Organic chemistry nomenclature: an online learning tool, Presented at the Canadian Network for Innovation in Education.
  13. Flynn A. B., Caron J., Laroche J., Daviau-Duguay M., Marcoux C. and Richard G., (2014), nomenclature101.com: A Free, Student-Driven Organic Chemistry Nomenclature Learning Tool, J. Chem. Educ., 91(11), 1855–1859, DOI: http://10.1021/ed500353a.
  14. González-Cruz J., Rodríguez-Sotres R. and Rodríguez-Penagos M., (2003), On the convenience of using a computer simulation to teach enzyme kinetics to undergraduate students with biological chemistry-related curricula, Biochem. Mol. Biol. Educ., 31(2), 93–101, DOI: http://10.1002/bmb.2003.494031020193.
  15. Guadagnoli E. and Velicer W. F., (1988), Relation to sample size to the stability of component patterns, Psychol. Bull., 103(2), 265–275, DOI: http://10.1037/0033-2909.103.2.265.
  16. Guskey T. R., (2002), Does It Make a Difference? Evaluating Professional Development, Educ. Leader., 59(6), 45–51.
  17. Guskey T. R., (2010), Professional Development and Teacher Change, Teachers Teach.: Theor. Pract., 8(3), 381–391, DOI: http://10.1080/135406002100000512.
  18. Harshman J. and Yezierski E., (2015), Guiding teaching with assessments: high school chemistry teachers' use of data-driven inquiry, Chem. Educ. Res. Pract., 16(1), 93–103, DOI: 10.1039/C4RP00188E.
  19. Hawker M. J., Dysleski L. and Rickey D., (2016), Investigating General Chemistry Students' Metacognitive Monitoring of Their Exam Performance by Measuring Postdiction Accuracies over Time, J. Chem. Educ., 93(5), 832–840, DOI: http://10.1021/acs.jchemed.5b00705.
  20. Kaiser H. F., (1974), An index of factorial simplicity, Psychometrika, 39(1), 31–36, DOI: http://10.1007/BF02291575.
  21. Karatjas A. G., (2013), Comparing College Students' Self-Assessment of Knowledge in Organic Chemistry to Their Actual Performance, J. Chem. Educ., 90(8), 1096–1099, DOI: http://10.1021/ed400037p.
  22. Kinnebrew J. and Biswas G., (2011), Modeling and Measuring Self-Regulated Learning in Teachable Agent Environments, J. e-Learn. Knowl. Soc., 7(2), 19–35.
  23. Kornell N., (2009), Optimising learning using flashcards: spacing is more effective than cramming, Appl. Cognit. Psychol., 23(9), 1297–1317, DOI: http://10.1002/acp.1537.
  24. Krippendorff K., (2004), Reliability in Content Analysis, Hum. Comm. Res., 30(3), 411–433, DOI: http://10.1111/j.1468-2958.2004.tb00738.x.
  25. Kruger J. and Dunning D., (1999), Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments, J. Pers. Soc. Psychol., 77(6), 1121–1134, DOI: http://10.1037/0022-3514.77.6.1121.
  26. Larsen R. and Warne R. T., (2010), Estimating confidence intervals for eigenvalues in exploratory factor analysis, Behav. Res. Meth., 42(3), 871–876, DOI: http://10.3758/BRM.42.3.871.
  27. Novak J. D., (2011), A theory of education: meaningful learning underlies the constructive integration of thinking, feeling, and acting leading to empowerment for commitment and responsibility, Meaningful Learning Review, 1(2), 1–14.
  28. Oriol M. D., Tumulty G. and Snyder K., (2010), Cognitive apprenticeship as a framework for teaching online, J. Online Learn. Teach., 6(1), 210–217.
  29. Patton M. Q., (2014), Qualitative Research and Evaluation Methods, 4 edn, Thousand Oaks, CA: SAGE Publications.
  30. Revelle W. and Zinbarg R. E., (2008), Coefficients Alpha, Beta, Omega and the glb: Comments on Sijtsma, Psychometrika, 74(1), 145–154, DOI: http://10.1007/s11336-008-9102-z.
  31. Top Hat, (2016), Top Hat. Tophat.com. Retrieved June 2016, from http://https://tophat.com/.
  32. Vygotsky L. S., (1978), in Cole M. Souberman E. John-Steiner V. and Scribner, S. (ed.), Mind in Society, Cambridge, MA: Harvard University Press.
This journal is © The Royal Society of Chemistry 2016