Keily
Heredia‡
,
Xiaoying
Xu‡
and
Jennifer E.
Lewis
*
Department of Chemistry, University of South Florida, 4202 E. Fowler Avenue CHE205A, Tampa, Florida 33620, USA. E-mail: jennifer@usf.edu; Fax: (813) 974-3203; Tel: (813) 974-1286
First published on 7th December 2011
The Particulate Nature of Matter and Chemical Bonding Diagnostic Instrument (Othman J., Treagust D. F. and Chandrasegaran A. L., (2008), Int. J. Sci. Educ., 30(11), 1531–1550) is used to investigate college students' understanding of two chemistry concepts: particulate nature of matter and chemical bonding. The instrument, originally developed for secondary school students, is a two-tier diagnostic test. In this study, the instrument was given as a paper and pencil test to college students enrolled in General Chemistry I during the second week of class in Spring 2010. General Chemistry I students were divided into three groups for analysis: (A) with preparatory chemistry, (B) repeating general chemistry (without preparatory chemistry); and (C) first time in general chemistry (without preparatory chemistry); Groups B and C are expected to have taken at least one year of secondary school chemistry. However, Group A had little or no exposure to high school chemistry. Regardless of their pathway into the course, students in General Chemistry I are expected to have a fair understanding of the two concepts. Furthermore, students' responses were used to think about the role of a preparatory chemistry course in promoting deep understanding of chemistry concepts. An ANCOVA analysis was performed. Preparatory Chemistry was found to have a statistically significant but small effect on students’ scores after controlling for prior math achievement. Confirmatory factor analysis and reliability studies were performed and are presented here. Analysis also revealed alternative conceptions.
Other approaches have been used to assess student learning. Qualitative approaches such as open-ended responses (Nyachwaya et al., 2011), and clinical interviews (Ebenezer and Gaskell, 1995; Costu, 2008), have been widely used as an effective tool to investigate students' thinking and conceptual understanding. While these approaches provide rich and detailed information, they are time consuming, and not easy to use in classroom assessment. Multiple-choice test format is convenient, and therefore, typically used in standardized and in classroom tests (Examinations Institute, 2011). For example, multiple-choice summative tests cover content as broadly as possible to reflect the student cumulative knowledge, and therefore, provide information about the student content knowledge at the end of a chapter or a semester in a particular subject area. However, the interpretation of each item in summative tests cannot specifically provide information about the alternative conceptions students have about chemistry concepts.
Chemistry-based diagnostic tests have been developed and used to measure student alternative conceptions (Treagust, 1986; Treagust, 1988; Voska and Heikkinen, 2000; Tsai and Chou, 2002; Treagust et al., 2011). These assessments can diagnose students' understanding of concepts that are included in the introductory college chemistry curriculum. Two important concepts, the particulate nature of matter and chemical bonding, are included in a diagnostic instrument developed by Othman et al. (2008) for secondary school students. Other instruments are available to measure students' alternative conceptions; however, they are often more general (Mulford and Robinson, 2002) or more appropriate for later in the curriculum (Villafañe et al., 2011). Because of its tight focus on two concepts and its suitability for students who have little experience with college chemistry, the two-tier Particulate Nature of Matter and Chemical Bonding Diagnostic Instrument was chosen for this study (Diagnostic Instrument hereafter). The two concept instrument is also accessible, short, and easy to administer, and its two-tier design allows students to provide both an answer and a reason.
The original study conducted by Othman et al. (2008) reported alternative conceptions about the particulate nature of matter and chemical bonding held by secondary school students in Singapore. More than half of the students in grades 9 (N = 140) and 10 (N = 120) believed that “an atom of an element shares some physical properties as a sample of the element.” Similar alternative conceptions about the particulate nature of matter have been found in different educational contexts, and reported in many other studies. For example, Taber (2001) and Johnson (2005) have explained that many students believe the particles in a substance possess the same macroscopic properties as the substance and suggested that instructional practices may reinforce this belief. Recently, Salta and Tzougraki (2011) found that secondary school students in Greece failed to successfully answer questions regarding the particulate nature of matter due to their inability to transfer understanding from the atomic level to the macroscopic level. A common alternative conception about chemical bonding found by Othman et al. (2008) was that “sodium chloride exists as molecules.” About 40% percent of the students in both grades selected this alternative conception. During one-on-one interviews with college students enrolled in introductory chemistry in the U.S., Teichert et al. (2008) also found that students believe that ionic compounds exist as molecules. Because similar alternative conceptions are so pervasive, appearing in students of different ages, in different countries, and through different methods of investigation, there is reason to check for these ideas among our students entering college level general chemistry.
It is common for science students to register for General Chemistry I during the fall semester, and register for General Chemistry II during the following spring semester. However, not every science major follows this path. First, there are often more students wanting to take General Chemistry I in the fall than seats available, and second, as described above, selected students are encouraged to take a preparatory chemistry course in the fall semester. There are also students who are able to enroll in General Chemistry I in the fall, but do not succeed with a course passing grade of C or above, and therefore cannot take General Chemistry II without repeating General Chemistry I. In order to meet student demand from these diverse pathways, General Chemistry I is routinely offered in the spring semester. Since the participants in this study are enrolled in the General Chemistry I course during the spring semester, it makes sense to examine the student population in more detail.
Three distinct groups emerge from this examination: Group A, the “with prep-chem” group, 364 students who have taken Preparatory Chemistry; Group B, the “repeaters”, 125 students who have not taken Preparatory Chemistry but who have been enrolled in General Chemistry I in a prior semester; and Group C, the “first-timers”, 236 students who have not taken Preparatory Chemistry but also have never been enrolled in General Chemistry I. Not only are these groups different in terms of their pathway to General Chemistry I in Spring 2010, they are also different with respect to demographic information. The prep-chem group is more heavily tipped toward female (68%) than male (32%) students as compared to the other two groups (χ2 (2, N = 725) = 20.7, p < 0.01) and has a more diverse population as compared to the other two groups, including a sizable number of underrepresented minority students (χ2 (12, N = 725) = 52.2, p < 0.01). In terms of prior math achievement, the prep-chem group, as expected, has a lower average Math SAT score than the other two groups (F (2, N = 576) = 22.6, p < 0.01). Detailed demographic information for each group is presented in the ESI.†
With Prep-Chem (N = 348) | Without Prep-Chem (N = 331) | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Item | Repeaters (N = 108) | First-timers (N = 223) | ||||||||||
A | B | C | ||||||||||
M | SD | Skew | Kurt | M | SD | Skew | Kurt | M | SD | Skew | Kurt | |
1 | 0.35 | 0.48 | 0.62 | −1.63 | 0.27 | 0.45 | 1.06 | −0.89 | 0.25 | 0.43 | 1.16 | −0.67 |
2 | 0.45 | 0.50 | 0.21 | −1.97 | 0.47 | 0.50 | 0.11 | −2.03 | 0.46 | 0.50 | 0.17 | −1.99 |
3 | 0.38 | 0.49 | 0.51 | −1.75 | 0.29 | 0.45 | 0.95 | −1.11 | 0.30 | 0.46 | 0.88 | −1.24 |
4 | 0.39 | 0.49 | 0.46 | −1.80 | 0.43 | 0.50 | 0.30 | −1.94 | 0.39 | 0.49 | 0.47 | −1.79 |
5 | 0.14 | 0.35 | 2.11 | 2.46 | 0.24 | 0.43 | 1.23 | −0.50 | 0.23 | 0.42 | 1.30 | −0.31 |
6 | 0.07 | 0.26 | 3.25 | 8.61 | 0.08 | 0.28 | 3.06 | 7.49 | 0.05 | 0.23 | 3.98 | 13.98 |
7 | 0.07 | 0.25 | 3.51 | 10.37 | 0.08 | 0.28 | 3.06 | 7.49 | 0.04 | 0.19 | 5.03 | 23.46 |
8 | 0.19 | 0.39 | 1.57 | 0.46 | 0.20 | 0.40 | 1.49 | 0.23 | 0.17 | 0.37 | 1.81 | 1.28 |
9 | 0.38 | 0.49 | 0.51 | −1.75 | 0.39 | 0.49 | 0.46 | −1.82 | 0.37 | 0.48 | 0.55 | −1.71 |
10 | 0.46 | 0.50 | 0.16 | −1.99 | 0.31 | 0.47 | 0.81 | −1.37 | 0.46 | 0.50 | 0.15 | −1.99 |
Total score | 2.87 | 1.75 | 0.50 | −0.28 | 2.77 | 1.76 | 0.55 | −0.30 | 2.71 | 1.75 | 0.60 | −0.15 |
Data analysis then moves to interpretation. To determine whether group membership was associated with a difference in performance on the Diagnostic Instrument, ANCOVA was performed in SAS 9.1. Detail information about the ANCOVA analysis is presented in the ESI.† Finally, chi-square comparisons allowed the identification of two alternative conceptions that are less prevalent in a particular group.
Following Othman et al. (2008), student response patterns were used to identify prevalent alternative conceptions, but, rather than applying an across-the-board cutoff of 10%, a variable percentage based on the potential for guessing associated with a given item serves as the cutoff. One way of handling the guessing effect associated with a multiple choice assessment is to decide that, for a guesser, all options are equally plausible. For an exam question that has four options, the chance of randomly guessing the correct answer would then be approximately 25%. However, for a two-tier item, if each question has four options, the chance of guessing the correct answer combination drops to approximately 6% (0.25 × 0.25).
Another way to check for evidence of relationships among items is to utilize factor analysis. Othman et al. (2008) proposes that the Diagnostic Instrument is divided into two concepts, the particulate nature of matter and chemical bonding. Therefore, since the sample size is sufficiently large (Hogarty et al., 2005; MacCallum et al., 1999) a confirmatory factor analysis (CFA) was performed in Mplus 5.2 on a first-order model with two latent factors that were allowed to correlate. According to the proposed model items 1, 2, 3, 4, and 5 were set to load in factor “Particulate Nature” only, and items 6, 7, 8, 9, and 10 were set to load in factor “Chemical Bonding” only. While model fit statistics (see ESI†) were within the range for an acceptable fit (Brown, 2006), the factor loadings were problematic. As shown in Table 2, all items except 1 and 3 load only weakly in their proposed factor. In addition, the loadings for items 7 and 8 from the Chemical Bonding concept are not large enough to be significant. A CFA for a 1-factor model, which places all 10 items in a single factor, was also performed. The fit statistics were slightly worse than for the 2-factor model, and factor loadings were very similar to the ones presented for the 2-factor model, so are not shown here. Like the reliability analysis, these factor analysis results again suggest the instrument is functioning more like a traditional content test in this setting, with student responses for one item on the test not necessarily related to their responses for a another item.
In summary, before the inclusion of SAT Math scores as a covariate, no statistically significant difference among groups was found. The ANCOVA, however, revealed that the Preparatory Chemistry course was associated with students' higher performance on the Diagnostic Instrument, having a statistically significant but small effect. Previous exposure to the General Chemistry I course had a positive though not significant effect on students' scores. This result aligns with common sense about the effect of prior coursework in chemistry, and provides hope that Preparatory Chemistry is assisting students to some degree. However, since the second tier of each item provides information about the student conceptual understanding, a detailed item-by-item look at the data will determine whether there are any particular alternative conceptions that appear to a lesser extent for the prep-chem group of students.
In this study, the number of response options for each question varies from two to five, so the chance of guessing the correct answer for both items under the equal plausibility assumption varies as well, but is never higher than 12.5%. Since the means for the three groups in this study, with preparatory chemistry (2.87), repeaters (2.77), and first-timers (2.71), are quite a bit lower than the mean of 5.2 reported by Othman et al. (2008), we were concerned that students may have been guessing in many cases. Given that, in the real world of guessing, not all response options are equally plausible, we chose a cutoff value of twice the approximate random guessing value. In this way, we can be certain that the response patterns we identify are very likely to be common alternative conceptions. In other words, if students choose a particular combination of response options for the two items at a level that is so much higher than by chance alone, we can say with confidence that there is something attractive about that combination.
The observed percent of students choosing alternative conceptions is compared with twice the approximate guessing value in Tables 3 through 6. We will use this comparison to guide us in the next phase of analysis, determining whether there is any evidence that a Preparatory Chemistry course helps to alleviate a prevalent misconception, but a descriptive look at the patterns is also revealing.
A total of eleven alternative conceptions related to the topic of Particulate Nature of Matter were found, and are shown in Tables 3 and 4. Five answer choice combinations were above the cutoff for all three groups of students, including, in order of decreasing frequency, the notion that an atom has the same properties as an element, that stirring is necessary for dissolution to occur, and that iodine weighs less in the vapor phase than in the solid phase. For phase changes in water, there is more diversity. Two answer choice combinations, that hydrogen and oxygen atoms in water molecules break apart to form gases and that evaporation can be thought of as water molecules escaping into the air with no particulate representation, are above the cutoff for all three groups. Six other combinations are more mixed, with not all groups showing evidence of the identical alternative conceptions. Choosing the correct particulate representation for evaporated water was, however, a problem for all three groups, as over 50% of each group got the first tier of item 3 wrong (see the ESI†). For three cases, in which students answered the first tier of item 1 correctly (that the bubbles in boiling water contain water vapor) but had trouble with the reason, the repeaters are slightly below the cutoff in two instances and the prep-chem students are slightly below for the third.
Alternative Conception | Choice combination | A: With Prep-Chem (N = 348) | B: Repeaters (N = 108) | C: First-timers (N = 223) | Twice % guessing value |
---|---|---|---|---|---|
Molecular and macroscopic properties | |||||
An atom is the smallest particle of an element that has the same properties as the element. | Item 5 [A1] | 49.4 | 43.5 | 42.6 | 12.5 |
Dissolving | |||||
A solute only dissolves when stirring causes the crystals to break into smaller particles that can no longer be seen. | Item 4 [B4] | 39.1 | 37.0 | 44.4 | 25 |
Conservation of matter during phase changes | |||||
Iodine gas weighs less than solid iodine. | Item 2 [A1] | 21.8 | 22.2 | 22.8 | 17 |
Boiling/evaporation of water | |||||
The hydrogen and oxygen atoms in water molecules break away from each other to form gaseous oxygen and hydrogen. | Item 1 [B1] | 13.2 | 23.1 | 14.0 | 8 |
Water molecules have escaped into the air, and are not represented in a particulate way. | Item 3 [C2] | 10.34 | 11.1 | 11.21 | 8 |
Alternative Conception | Choice combination | A: With Prep-Chem (N = 348) | B: Repeaters (N = 108) | C: First-timers (N = 223) | Twice % guessing value |
---|---|---|---|---|---|
a A bold number is used to indicate being above the cutoff in the last column. | |||||
Boiling/evaporation of water | |||||
Water molecules have broken free from one another and decomposed into oxygen and hydrogen atoms. | Item 3 [D4] | 9.18 | 7.41 | 14.8 | 8 |
Water molecules have decomposed into diatomic oxygen and hydrogen gas. | Item 3 [A3] | 8.33 | 11.1 | 4.93 | 8 |
Water molecules have decomposed into oxygen atoms and hydrogen atoms. | Item 3 [D1] | 3.45 | 9.25 | 5.83 | 8 |
When the water is heated, the air between the water molecules is released in the form of bubbles. | Item 1 [D2] | 9.70 | 5.56 | 10.3 | 8 |
The hydrogen and oxygen atoms in water molecules break away from each other to form gaseous water vapor. | Item 1 [D1] | 8.90 | 7.41 | 10.8 | 8 |
Heat energy is absorbed by the water and released as bubbles. | Item 1 [D3] | 6.90 | 12.0 | 13.5 | 8 |
A total of seven alternative conceptions related to the topic of Chemical Bonding were found, and are shown in Tables 5 and 6. Four alternative conceptions were found in all three groups of students, including that molten calcium fluoride's free electrons allow it to conduct electricity, and that sodium chloride contains molecules formed by the donation of a valence electron from sodium to chlorine. The two other answer combinations have opposite perspectives on whether carbon dioxide has low melting and boiling points but the commonality of using an empirical fact (carbon dioxide is a gas at room temperature) rather than a particulate-level explanation as a reason. Two alternative conceptions were found among first-timers only, that sodium chloride produces free electrons when dissolved in water and that calcium fluoride is made up of covalent molecules. Finally, the alternative conception that sodium chloride is made up of covalent molecules involving a shared pair of electrons is prevalent for repeaters and first-timers, but the students with prep-chem were not, as a group, quite as attracted to this idea.
Alternative Conception | Choice combination | A: With Prep-Chem (N = 348) | B: Repeaters (N = 108) | C: First-timers (N = 223) | Twice % guessing value |
---|---|---|---|---|---|
Electrical conductivity of ionic compounds | |||||
Calcium fluoride is an ionic compound; it has free electrons that enable it to conduct electricity. | Item 8 [A3] | 42.5 | 31.5 | 33.2 | 25 |
Structure of sodium chloride | |||||
After donating its valence electron to the chloride, the sodium ions form a molecule with the chloride ion. | Item 6 [A2] | 39.1 | 39.8 | 33.2 | 25 |
Intermolecular and intramolecular forces | |||||
Carbon dioxide has low melting and boiling points because is a gas at room temperature. | Item 7 [A4] | 29.9 | 38.9 | 41.3 | 25 |
Carbon dioxide doesn’t have low melting and boiling points because is a gas at room temperature. | Item 7 [B4] | 28.5 | 27.7 | 34.5 | 25 |
Alternative Conception | Choice combination | A: With Prep-Chem (N = 348) | B: Repeaters (N = 108) | C: First-timers (N = 223) | Twice % guessing value |
---|---|---|---|---|---|
a A bold number is used to indicate being above the cutoff in the last column. | |||||
Electrical conductivity of ionic compounds | |||||
NaCl produces free electrons when dissolved in water, but not when is solid. | Item 9 [A3] | 24.7 | 19.4 | 26.0 | 25 |
Calcium fluoride consists of covalent molecules. | Item 8 [B4] | 18.9 | 22.2 | 30.0 | 25 |
Structure of sodium chloride | |||||
The sodium atom shares a pair of electrons with the chlorine atom to form a simple molecule. | Item 6 [A1] | 21.6 | 32.4 | 33.6 | 25 |
Since we are interested specifically in the role Preparatory Chemistry is supposed to play in helping students understand concepts from secondary school chemistry before they enter General Chemistry I, we focus on the two cases from the descriptive analysis in which the cutoff indicated an alternative conception for the other two groups, but not for the students with prep-chem. The question is whether the percentage of students with the alternative conception is significantly different among the three groups, which can be addressed with a chi-square test. For the alternative conception that heat energy is absorbed by boiling water and released as bubbles, the percentages were found to be significantly different (χ2 (2, N = 679) = 7.25, p = 0.03), with prep-chem significantly lower. For the alternative conception that sodium chloride is made up of covalent molecules involving a shared pair of electrons, again the percentages were found to be significantly different (χ2 (2, N = 679) = 11.00, p = 0.003) with prep-chem significantly lower.
While these statistical tests provide some evidence that Preparatory Chemistry students lack two alternative ideas that the other two groups of students still have, when the curriculum for Preparatory Chemistry is examined, it is not at all clear by what mechanism these two particular alternative conceptions would have been corrected while the other alternative conceptions persisted. As indicated by all three groups' responses to the Diagnostic Instrument, the dismaying news is that a high percentage of all students entering General Chemistry I in Spring 2010 had not yet understood fundamental ideas of the particle theory, even though one group of them took a course that, on the surface, contained exactly the right level of content knowledge for them to be successful on this instrument.
Two tier multiple-choice instruments like the one used in this study are good formative assessment tools. The use of a second tier (reasoning) provides instructors with more information about student conceptual understanding. The next step, once alternative conceptions have been uncovered, is to determine whether instructional strategies can improve student understanding. These types of assessments then become useful as evaluative resources to examine the effectiveness of instruction in addressing student alternative conceptions.
In terms of the role of Preparatory Chemistry, the ANCOVA results, although statistically significant, showed that the course effect on the overall Diagnostic Instrument score is small. In addition, we were able to find only two alternative conceptions that were less attractive to students with prep-chem as compared to the other two groups. Therefore, it is difficult to conclude that exposure to Preparatory Chemistry helped students' understanding of the particulate nature of matter and chemical bonding in a substantive way. The results of this assessment must be used to rethink the role of that course. If the course intention remains to help students to understand and apply the particulate nature of matter as described in the syllabus, specifically targeting the alternative conceptions found in this study can help.
A contribution of this study has been the implementation of a cutoff derived from a plausible guessing frequency to identify students' alternative conceptions. An approximate “guessing percentage” for each item was calculated so that response patterns that were the result of guessing alone could be reduced to the level of noise. Using twice the approximate guessing value as a cutoff, we were able to identify very common alternative conceptions in the study population.
One of the limitations of this study includes using a low stakes exam to evaluate students' alternative conceptions. Although the Diagnostic Instrument was given under similar conditions as the regular course exams, students knew that it was a low stakes exam. This can influence the performance of the students, reducing the impetus to think hard about subtle differences among response options. Another limitation is that the Diagnostic Instrument was not given again to the students at a later point in the term. Therefore, we do not have any data about whether the students were able to overcome or change the alternative conceptions described in this paper. This next step will be particularly important in the future, to determine to what degree the General Chemistry I course instructors are able to use these results to inform their teaching.
Using this instrument to try to understand the role of Preparatory Chemistry has left us with some outstanding questions for future research. We did identify specific answer combinations that were prevalent in all three populations, but, in the absence of interviews or open response items, we feel unsatisfied about the robustness of these alternative conceptions. Would students have responded in the same way if the questions were phrased slightly differently, or if they were asked to explain their thinking? Constrained by a very large chemistry course-taking population and responsible to instructors who value having information about all students, we do need a multiple-choice approach. While the two-tier Diagnostic Instrument has a significant advantage over a single-item test, we believe there is room for still more refinement. The number of items related to each specific concept within the larger topics of Particulate Nature of Matter and Chemical Bonding limited our understanding with regard to the students' understanding of the specifics. Having multiple items designed to probe for the same misconception would provide a greater certainty that students are answering as they truly think, and are not overly influenced by a small detail of question wording or a momentary distraction. What this study has highlighted for us is the critical importance of having even shorter, more-tightly-focused instruments that help instructors probe particular alternative conceptions at the appropriate time in the curriculum.
The Diagnostic Instrument was created to assess students' understanding about two broad concepts, particulate nature of matter and chemical bonding. It was also intended to identify whether students' understanding about one of the concepts influences their understanding of the other concept. Results from the CFA in this case did not support a two-factor structure, so we were unable to use the instrument for that purpose within this study, and we look forward to others' work on this topic. Regardless, the two-tier Diagnostic Instrument provided useful information related to students' conceptions about the particulate nature of matter and chemical bonding that poses challenges for the courses investigated in this study.
Footnotes |
† Electronic supplementary information (ESI) available. See DOI: 10.1039/c0rp90017f |
‡ The two authors contributed equally to this work. |
This journal is © The Royal Society of Chemistry 2012 |