View PDF VersionPrevious ArticleNext Article
 
DOI: 10.1039/C0RP90017F (Paper) Chem. Educ. Res. Pract., 2012, 13, 30-38

The application and evaluation of a two-concept diagnostic instrument with students entering college general chemistry†

Keily Heredia‡ , Xiaoying Xu‡ and Jennifer E. Lewis *
Department of Chemistry, University of South Florida, 4202 E. Fowler Avenue CHE205A, Tampa, Florida 33620, USA. E-mail: jennifer@usf.edu; Fax: (813) 974-3203; Tel: (813) 974-1286

Received 30th October 2010 , Accepted 23rd October 2011

First published on 7th December 2011

Abstract

The Particulate Nature of Matter and Chemical Bonding Diagnostic Instrument (Othman J., Treagust D. F. and Chandrasegaran A. L., (2008), Int. J. Sci. Educ., 30(11), 1531–1550) is used to investigate college students' understanding of two chemistry concepts: particulate nature of matter and chemical bonding. The instrument, originally developed for secondary school students, is a two-tier diagnostic test. In this study, the instrument was given as a paper and pencil test to college students enrolled in General Chemistry I during the second week of class in Spring 2010. General Chemistry I students were divided into three groups for analysis: (A) with preparatory chemistry, (B) repeating general chemistry (without preparatory chemistry); and (C) first time in general chemistry (without preparatory chemistry); Groups B and C are expected to have taken at least one year of secondary school chemistry. However, Group A had little or no exposure to high school chemistry. Regardless of their pathway into the course, students in General Chemistry I are expected to have a fair understanding of the two concepts. Furthermore, students' responses were used to think about the role of a preparatory chemistry course in promoting deep understanding of chemistry concepts. An ANCOVA analysis was performed. Preparatory Chemistry was found to have a statistically significant but small effect on students’ scores after controlling for prior math achievement. Confirmatory factor analysis and reliability studies were performed and are presented here. Analysis also revealed alternative conceptions.

Introduction

College-level chemistry courses are required for science majors. Students who are unable to successfully pass introductory college chemistry are prevented from continuing in science-oriented programs. Therefore, student performance in introductory college-level chemistry courses remains a recognized area of concern. Extensive research has been done to predict student performance in college chemistry courses (Russell, 1994; Wagner et al., 2002; Tai et al., 2005; Potgieter et al., 2010). Previous studies have used the California Chemistry Diagnostic Exam (McFate and Olmsted III, 1999), the SAT (Spencer, 1996), as well as logical reasoning instruments (Bunce and Hutchinson, 1993; Lewis and Lewis, 2007; Jiang et al., 2010) to predict student achievement in introductory college chemistry.

Other approaches have been used to assess student learning. Qualitative approaches such as open-ended responses (Nyachwaya et al., 2011), and clinical interviews (Ebenezer and Gaskell, 1995; Costu, 2008), have been widely used as an effective tool to investigate students' thinking and conceptual understanding. While these approaches provide rich and detailed information, they are time consuming, and not easy to use in classroom assessment. Multiple-choice test format is convenient, and therefore, typically used in standardized and in classroom tests (Examinations Institute, 2011). For example, multiple-choice summative tests cover content as broadly as possible to reflect the student cumulative knowledge, and therefore, provide information about the student content knowledge at the end of a chapter or a semester in a particular subject area. However, the interpretation of each item in summative tests cannot specifically provide information about the alternative conceptions students have about chemistry concepts.

Chemistry-based diagnostic tests have been developed and used to measure student alternative conceptions (Treagust, 1986; Treagust, 1988; Voska and Heikkinen, 2000; Tsai and Chou, 2002; Treagust et al., 2011). These assessments can diagnose students' understanding of concepts that are included in the introductory college chemistry curriculum. Two important concepts, the particulate nature of matter and chemical bonding, are included in a diagnostic instrument developed by Othman et al. (2008) for secondary school students. Other instruments are available to measure students' alternative conceptions; however, they are often more general (Mulford and Robinson, 2002) or more appropriate for later in the curriculum (Villafañe et al., 2011). Because of its tight focus on two concepts and its suitability for students who have little experience with college chemistry, the two-tier Particulate Nature of Matter and Chemical Bonding Diagnostic Instrument was chosen for this study (Diagnostic Instrument hereafter). The two concept instrument is also accessible, short, and easy to administer, and its two-tier design allows students to provide both an answer and a reason.

Study purpose

The purpose of this study is to probe student understanding of two important topics from secondary school chemistry that will be covered in greater depth in introductory college-level chemistry, and to compare the understanding of students who enter the course via different pathways. Within the context of the study, students who enroll directly in General Chemistry I are expected to have taken a year of chemistry in secondary school. However, some students in the U.S. are able to graduate from secondary school without taking a course focused on chemistry. For example, to achieve the three required high school science credits (full-year courses) in the school district in which the university is located, students may take an introductory integrated science course in which some chemistry topics may play a role, followed by a biological science course, capped with an elective such as ecology. Therefore, upon their arrival at the university, students without (or with limited exposure to) secondary school chemistry are recommended to take Preparatory Chemistry prior to General Chemistry I. Regardless of the pathway into General Chemistry I, students need to have developed a conceptual understanding of the particulate nature of matter and chemical bonding before enrolling in the course. We believe that the Particulate Nature of Matter and Chemical Bonding Diagnostic Instrument is suitable for determining the state of their understanding at the time of course entry. However, one important question is whether the instrument functions well at the college level. Thus, this study will begin by investigating the factor structure of the item scores to determine whether the instrument's designed factors are relevant for the present use. The study will then move to the interpretation of the students' responses to examine whether a preparatory chemistry course appears to have any influence on student understanding of the two topics.

Research questions

The following research questions will guide this study: (1) Do students entering General Chemistry I having taken Preparatory Chemistry perform better on the Diagnostic Instrument than students without Preparatory Chemistry? (2) Which alternative conceptions, if any, do students entering General Chemistry I have? (3) Are the alternative conceptions the same or different for students entering General Chemistry I with Preparatory Chemistry as compared to those entering without it?

Course context and chosen instrument

Preparatory chemistry

Preparatory Chemistry is a one-semester course that is recommended for students with an SAT Math score lower than 550, and for those who did not take secondary school chemistry. The course is intended to prepare students to take General Chemistry I, and has a history of promoting success in that course (Garcia, 2010). According to the syllabus, the emphasis of Preparatory Chemistry is on providing foundational understanding of chemical principles and developing fundamental processing skills such as critical thinking and learning strategies. Upon completion of the course, students are expected to understand and be able to apply the particulate nature of matter and the first law of thermodynamics. More detailed course information is presented in the ESI.†

General chemistry I

General Chemistry I is the first of two semester-long introductory-level college chemistry courses, required for all science majors. Students who take General Chemistry I without taking Preparatory Chemistry are expected to have at least one year of secondary school chemistry and evidence of prior mathematics achievement, such as an SAT Math score of at least 550 or a passing grade in a college-level algebra course. Principles and applications of chemistry, including properties of substances and reactions, thermo-chemistry, atomic-molecular structure and bonding, and periodic properties of elements and compounds are discussed in the course. The structure of the course has been discussed previously (Lewis and Lewis, 2005, 2008).

Diagnostic instrument

The Particulate Nature of Matter and Chemical Bonding Diagnostic Instrument (Othman et al., 2008), was originally developed for secondary school students. It is a two-tier diagnostic used to investigate students' conceptions of the particulate nature of matter, and their understanding of chemical bonding. The instrument consists of ten items, five items for each concept. These items comprised topics such as molecular and macroscopic properties, solutions, conservation of matter, and phase changes for the concept of particulate nature of matter, and electrical conductivity, structure of sodium chloride, and intermolecular forces for the concept of chemical bonding. Each item consists of two multiple-choice questions. The first tier of each item is a multiple-choice question that relates to a problem statement. The second tier of each item is composed of a set of explanations for the answers from the first tier. The second tier (reasoning) can provide information about which alternative conceptions students have about the two chemistry concepts.

The original study conducted by Othman et al. (2008) reported alternative conceptions about the particulate nature of matter and chemical bonding held by secondary school students in Singapore. More than half of the students in grades 9 (N = 140) and 10 (N = 120) believed that “an atom of an element shares some physical properties as a sample of the element.” Similar alternative conceptions about the particulate nature of matter have been found in different educational contexts, and reported in many other studies. For example, Taber (2001) and Johnson (2005) have explained that many students believe the particles in a substance possess the same macroscopic properties as the substance and suggested that instructional practices may reinforce this belief. Recently, Salta and Tzougraki (2011) found that secondary school students in Greece failed to successfully answer questions regarding the particulate nature of matter due to their inability to transfer understanding from the atomic level to the macroscopic level. A common alternative conception about chemical bonding found by Othman et al. (2008) was that “sodium chloride exists as molecules.” About 40% percent of the students in both grades selected this alternative conception. During one-on-one interviews with college students enrolled in introductory chemistry in the U.S., Teichert et al. (2008) also found that students believe that ionic compounds exist as molecules. Because similar alternative conceptions are so pervasive, appearing in students of different ages, in different countries, and through different methods of investigation, there is reason to check for these ideas among our students entering college level general chemistry.

Diagnostic instrument & preparatory chemistry

In order to be successful in General Chemistry I, students need to have a conceptual understanding of the particulate nature of matter and chemical bonding. A closer look at the Preparatory Chemistry syllabus, textbook (Tro, 2009), and in-class activities revealed that students discussed matter, compounds, solutions, and physical changes as early as the second and third week of class. The concepts are then built on this general overview, and become more in-depth in later weeks. By the fourth and fifth week of classes, students are challenged with questions about ionic and molecular compounds. Various examples using macro and micro representations of NaCl and CO2 are used to explain the difference between ionic and molecular compounds. In week seven, students discuss the concept of solubility. A sodium chloride solution, represented as NaCl(aq) is provided as an example to discuss electrical conductivity. By the thirteenth and fourteenth week of class, students review in further detail the concept of phase changes. A picture with a pot of water is shown, and an explanation about how the bubbles are formed is provided. Multiple representations of NaCl, CO2, and H2O are used throughout each chapter in the book to encourage student conceptual understanding. At the end of each chapter, students are confronted with mathematical problems, which assumes that students have learned the concepts. After considering the topics discussed in Preparatory Chemistry, one would expect that this group of students would have a better conceptual understanding of the two concepts measured by the Diagnostic Instrument than would students entering from diverse secondary school chemistry backgrounds.

Methodology

Participants

This study took place at a large public research university in the southeastern United States. The instrument was administered to students enrolled in General Chemistry I in Spring 2010, during the second week of class as a paper and pencil test. The students were given 25 min to answer the 10-item Diagnostic Instrument. Students in this course were from more than 25 majors, the largest of which were (1) bio-medical sciences 26%, (2) biology 17%, and (3) pre-medical sciences 10%. About 3% of the students were majoring in chemistry. This diverse sample is typical of the student population taking introductory college-level chemistry at this institution.

It is common for science students to register for General Chemistry I during the fall semester, and register for General Chemistry II during the following spring semester. However, not every science major follows this path. First, there are often more students wanting to take General Chemistry I in the fall than seats available, and second, as described above, selected students are encouraged to take a preparatory chemistry course in the fall semester. There are also students who are able to enroll in General Chemistry I in the fall, but do not succeed with a course passing grade of C or above, and therefore cannot take General Chemistry II without repeating General Chemistry I. In order to meet student demand from these diverse pathways, General Chemistry I is routinely offered in the spring semester. Since the participants in this study are enrolled in the General Chemistry I course during the spring semester, it makes sense to examine the student population in more detail.

Three distinct groups emerge from this examination: Group A, the “with prep-chem” group, 364 students who have taken Preparatory Chemistry; Group B, the “repeaters”, 125 students who have not taken Preparatory Chemistry but who have been enrolled in General Chemistry I in a prior semester; and Group C, the “first-timers”, 236 students who have not taken Preparatory Chemistry but also have never been enrolled in General Chemistry I. Not only are these groups different in terms of their pathway to General Chemistry I in Spring 2010, they are also different with respect to demographic information. The prep-chem group is more heavily tipped toward female (68%) than male (32%) students as compared to the other two groups (χ2 (2, N = 725) = 20.7, p < 0.01) and has a more diverse population as compared to the other two groups, including a sizable number of underrepresented minority students (χ2 (12, N = 725) = 52.2, p < 0.01). In terms of prior math achievement, the prep-chem group, as expected, has a lower average Math SAT score than the other two groups (F (2, N = 576) = 22.6, p < 0.01). Detailed demographic information for each group is presented in the ESI.†

Data analysis

When data from any instrument is collected, the first step is to examine descriptive statistics. SPSS 18.0 was used to obtain descriptive statistics for the ten items and total score on the diagnostic instrument, and the results appear in Table 1. The next step is to look for reliability and validity evidence. For reliability, Cronbach's alpha values were calculated in SAS 9.1, and the results compared to the commonly used cutoff of 0.70 (Thompson, 2003; Murphy and Davidshofer, 2005). For factorial validity, a confirmatory factor analysis in Mplus 5.2 estimated how well the 2-concept model for the instrument fit the item data obtained with our sample (Crocker and Algina, 2006). In addition to recommended cutoffs for model fit statistics (Hu and Bentler, 1999; Brown, 2006), factor correlation and item loadings are used to evaluate the fit.
Table 1 Descriptive statistics of item score and total score (N = 679)
  With Prep-Chem (N = 348) Without Prep-Chem (N = 331)
Item   Repeaters (N = 108) First-timers (N = 223)
A B C
  M SD Skew Kurt M SD Skew Kurt M SD Skew Kurt
1 0.35 0.48 0.62 −1.63 0.27 0.45 1.06 −0.89 0.25 0.43 1.16 −0.67
2 0.45 0.50 0.21 −1.97 0.47 0.50 0.11 −2.03 0.46 0.50 0.17 −1.99
3 0.38 0.49 0.51 −1.75 0.29 0.45 0.95 −1.11 0.30 0.46 0.88 −1.24
4 0.39 0.49 0.46 −1.80 0.43 0.50 0.30 −1.94 0.39 0.49 0.47 −1.79
5 0.14 0.35 2.11 2.46 0.24 0.43 1.23 −0.50 0.23 0.42 1.30 −0.31
6 0.07 0.26 3.25 8.61 0.08 0.28 3.06 7.49 0.05 0.23 3.98 13.98
7 0.07 0.25 3.51 10.37 0.08 0.28 3.06 7.49 0.04 0.19 5.03 23.46
8 0.19 0.39 1.57 0.46 0.20 0.40 1.49 0.23 0.17 0.37 1.81 1.28
9 0.38 0.49 0.51 −1.75 0.39 0.49 0.46 −1.82 0.37 0.48 0.55 −1.71
10 0.46 0.50 0.16 −1.99 0.31 0.47 0.81 −1.37 0.46 0.50 0.15 −1.99
Total score 2.87 1.75 0.50 −0.28 2.77 1.76 0.55 −0.30 2.71 1.75 0.60 −0.15

Data analysis then moves to interpretation. To determine whether group membership was associated with a difference in performance on the Diagnostic Instrument, ANCOVA was performed in SAS 9.1. Detail information about the ANCOVA analysis is presented in the ESI.† Finally, chi-square comparisons allowed the identification of two alternative conceptions that are less prevalent in a particular group.

Guessing value

Although the strength of a multiple choice assessment is that it can be given to a large number of students, as in this setting, a weakness is that is hard to identify whether students have a real misconception or are just guessing. With smaller groups, interviews and/or open-response options allow for additional analysis, but with over 500 students in a sample, the workload for these two strategies becomes prohibitively large, and clever ways of using multiple-choice instruments are needed.

Following Othman et al. (2008), student response patterns were used to identify prevalent alternative conceptions, but, rather than applying an across-the-board cutoff of 10%, a variable percentage based on the potential for guessing associated with a given item serves as the cutoff. One way of handling the guessing effect associated with a multiple choice assessment is to decide that, for a guesser, all options are equally plausible. For an exam question that has four options, the chance of randomly guessing the correct answer would then be approximately 25%. However, for a two-tier item, if each question has four options, the chance of guessing the correct answer combination drops to approximately 6% (0.25 × 0.25).

Results and discussion

For the Spring 2010 data, 683 responses were returned. Four sets of responses could not be verified as from students enrolled in the course, and were therefore excluded from this analysis for a total of 679 students. For the remaining 679 students, the item mean scores range from 0.06 (6%) to 0.46 (46%), and standard deviations range from 0.24 to 0.50. Descriptive statistics for each of the three groups are shown in Table 1. Items 6 and 7 are extremely difficult for all three groups, with less than 10% of students answering correctly in each case. The large values (larger than 1) for the skewness and kurtosis for these two items suggest violation of normality, so interpretations based solely on these items must be cautious. The overall mean (out of 10 items) of students without Preparatory Chemistry (both repeaters and first-timers) is 2.71, which breaks down as 2.77 for repeaters and 2.71 for first-timers. For students with Preparatory Chemistry the mean is 2.87. Overall, these low means suggest that students have a poor understanding of the two chemistry concepts regardless of their pathway into the course.

Reliability and validity

The reliability of the Diagnostic Instrument scores was examined using Cronbach's alpha estimates. A Cronbach's alpha of 0.62 is obtained when students' responses for the 20 questions are considered, which is close to the value of 0.66 reported in the literature (Othman et al., 2008) and not too far below the usual benchmark of 0.7 (Murphy and Davidshofer, 2005). Since this is a two-tier diagnostic instrument, two questions create an item for a total of 10 items. Therefore, one would expect to calculate a reliability coefficient based on 10 item scores rather than on 20 question scores. In that case a Cronbach's alpha of 0.42 is produced, which is considerably lower than cited in the literature. Splitting the instrument into its two suggested factors results in even lower reliability coefficients. Overall, these reliability investigations suggest that, for this sample, it is prudent to avoid making claims beyond the item level. In other words, from a reliability perspective, the instrument appears to be functioning similarly to an end-of-chapter test covering different aspects of related topics.

Another way to check for evidence of relationships among items is to utilize factor analysis. Othman et al. (2008) proposes that the Diagnostic Instrument is divided into two concepts, the particulate nature of matter and chemical bonding. Therefore, since the sample size is sufficiently large (Hogarty et al., 2005; MacCallum et al., 1999) a confirmatory factor analysis (CFA) was performed in Mplus 5.2 on a first-order model with two latent factors that were allowed to correlate. According to the proposed model items 1, 2, 3, 4, and 5 were set to load in factor “Particulate Nature” only, and items 6, 7, 8, 9, and 10 were set to load in factor “Chemical Bonding” only. While model fit statistics (see ESI†) were within the range for an acceptable fit (Brown, 2006), the factor loadings were problematic. As shown in Table 2, all items except 1 and 3 load only weakly in their proposed factor. In addition, the loadings for items 7 and 8 from the Chemical Bonding concept are not large enough to be significant. A CFA for a 1-factor model, which places all 10 items in a single factor, was also performed. The fit statistics were slightly worse than for the 2-factor model, and factor loadings were very similar to the ones presented for the 2-factor model, so are not shown here. Like the reliability analysis, these factor analysis results again suggest the instrument is functioning more like a traditional content test in this setting, with student responses for one item on the test not necessarily related to their responses for a another item.

Table 2 CFA loadings for the 2-factor solution (N = 679)
Item Loading
a Not significant.
Particulate nature of matter
Item 1 0.78
Item 2 0.29
Item 3 0.88
Item 4 0.20
Item 5 0.27
Chemical bonding
Item 6 0.40
Item 7 −0.16a
Item 8 0.05a
Item 9 0.42
Item 10 0.44

Comparison of students' performance with and without preparatory chemistry

At first glance, although students in Group A (with prep-chem) performed slightly better than students in the other two groups (Table 1) on the Diagnostic Instrument, the difference is quite small, such that ANOVA did not find a statistically significant difference among groups (F (2, N = 679) = 0.63, p = 0.53). However, the three groups are different in other ways. The students in Group A had significantly lower prior achievement in math than the other two groups. This is particularly important since in the study setting, prior math achievement as measured by SAT Math score has been strongly associated with chemistry content measures (Lewis and Lewis, 2007). Therefore, to further investigate the student performance on the Diagnostic Instrument, an ANCOVA was performed (Stevens, 1999). This analysis allowed us to determine whether the inclusion of SAT Math score as a covariate would reveal a significant pathway effect (see the ESI†). The main effect of the grouping variable was significant (F (3, N = 544) = 3.73, p = 0.02), yielding adjusted group means of 2.93, 2.65 and 2.45 for Groups A, B, and C respectively. In other words, after controlling for SAT Math, students who took the prep-chem course (Group A) are predicted to score 0.48 points higher on the Diagnostic Instrument than the students who simply waited to take General Chemistry I (the first-timers, or Group C). Although the repeaters (Group B) are predicted to score 0.20 points higher than first-timers (Group C) in the course, this difference is not significant. A graphical display of the ANCOVA analysis is provided in the ESI.†

In summary, before the inclusion of SAT Math scores as a covariate, no statistically significant difference among groups was found. The ANCOVA, however, revealed that the Preparatory Chemistry course was associated with students' higher performance on the Diagnostic Instrument, having a statistically significant but small effect. Previous exposure to the General Chemistry I course had a positive though not significant effect on students' scores. This result aligns with common sense about the effect of prior coursework in chemistry, and provides hope that Preparatory Chemistry is assisting students to some degree. However, since the second tier of each item provides information about the student conceptual understanding, a detailed item-by-item look at the data will determine whether there are any particular alternative conceptions that appear to a lesser extent for the prep-chem group of students.

Alternative conceptions

As discussed, the instrument seems to be functioning similarly to a focused content test in this setting. The best approach is to look at total score, as above, followed by performance on individual items. At the question level, students tend to perform better on the content part of an item (first tier) than on the reasoning part (second tier), and getting both parts of an item correct is difficult. These results are consistent with those reported by Othman et al. (2008). Students may know the correct answer for the content question being asked but have not understood why it is so, indicating a lack of understanding of the concept. These findings support the idea that it is appropriate to look at student response patterns within an item when deciding whether alternative conceptions are present. How to do so requires some decisions.

In this study, the number of response options for each question varies from two to five, so the chance of guessing the correct answer for both items under the equal plausibility assumption varies as well, but is never higher than 12.5%. Since the means for the three groups in this study, with preparatory chemistry (2.87), repeaters (2.77), and first-timers (2.71), are quite a bit lower than the mean of 5.2 reported by Othman et al. (2008), we were concerned that students may have been guessing in many cases. Given that, in the real world of guessing, not all response options are equally plausible, we chose a cutoff value of twice the approximate random guessing value. In this way, we can be certain that the response patterns we identify are very likely to be common alternative conceptions. In other words, if students choose a particular combination of response options for the two items at a level that is so much higher than by chance alone, we can say with confidence that there is something attractive about that combination.

The observed percent of students choosing alternative conceptions is compared with twice the approximate guessing value in Tables 3 through 6. We will use this comparison to guide us in the next phase of analysis, determining whether there is any evidence that a Preparatory Chemistry course helps to alleviate a prevalent misconception, but a descriptive look at the patterns is also revealing.

A total of eleven alternative conceptions related to the topic of Particulate Nature of Matter were found, and are shown in Tables 3 and 4. Five answer choice combinations were above the cutoff for all three groups of students, including, in order of decreasing frequency, the notion that an atom has the same properties as an element, that stirring is necessary for dissolution to occur, and that iodine weighs less in the vapor phase than in the solid phase. For phase changes in water, there is more diversity. Two answer choice combinations, that hydrogen and oxygen atoms in water molecules break apart to form gases and that evaporation can be thought of as water molecules escaping into the air with no particulate representation, are above the cutoff for all three groups. Six other combinations are more mixed, with not all groups showing evidence of the identical alternative conceptions. Choosing the correct particulate representation for evaporated water was, however, a problem for all three groups, as over 50% of each group got the first tier of item 3 wrong (see the ESI†). For three cases, in which students answered the first tier of item 1 correctly (that the bubbles in boiling water contain water vapor) but had trouble with the reason, the repeaters are slightly below the cutoff in two instances and the prep-chem students are slightly below for the third.

Table 3 Alternative conceptions about the Particulate Nature of Matter held by all three groups of General Chemistry I students (N = 679)
Alternative Conception Choice combination A: With Prep-Chem (N = 348) B: Repeaters (N = 108) C: First-timers (N = 223) Twice % guessing value
Molecular and macroscopic properties
An atom is the smallest particle of an element that has the same properties as the element. Item 5 [A1] 49.4 43.5 42.6 12.5
Dissolving
A solute only dissolves when stirring causes the crystals to break into smaller particles that can no longer be seen. Item 4 [B4] 39.1 37.0 44.4 25
Conservation of matter during phase changes
Iodine gas weighs less than solid iodine. Item 2 [A1] 21.8 22.2 22.8 17
Boiling/evaporation of water
The hydrogen and oxygen atoms in water molecules break away from each other to form gaseous oxygen and hydrogen. Item 1 [B1] 13.2 23.1 14.0 8
Water molecules have escaped into the air, and are not represented in a particulate way. Item 3 [C2] 10.34 11.1 11.21 8

Table 4 Alternative conceptions about Particulate Nature of Matter held by at least one group of General Chemistry I students (N = 679)a
Alternative Conception Choice combination A: With Prep-Chem (N = 348) B: Repeaters (N = 108) C: First-timers (N = 223) Twice % guessing value
a A bold number is used to indicate being above the cutoff in the last column.
Boiling/evaporation of water
Water molecules have broken free from one another and decomposed into oxygen and hydrogen atoms. Item 3 [D4] 9.18 7.41 14.8 8
Water molecules have decomposed into diatomic oxygen and hydrogen gas. Item 3 [A3] 8.33 11.1 4.93 8
Water molecules have decomposed into oxygen atoms and hydrogen atoms. Item 3 [D1] 3.45 9.25 5.83 8
When the water is heated, the air between the water molecules is released in the form of bubbles. Item 1 [D2] 9.70 5.56 10.3 8
The hydrogen and oxygen atoms in water molecules break away from each other to form gaseous water vapor. Item 1 [D1] 8.90 7.41 10.8 8
Heat energy is absorbed by the water and released as bubbles. Item 1 [D3] 6.90 12.0 13.5 8

A total of seven alternative conceptions related to the topic of Chemical Bonding were found, and are shown in Tables 5 and 6. Four alternative conceptions were found in all three groups of students, including that molten calcium fluoride's free electrons allow it to conduct electricity, and that sodium chloride contains molecules formed by the donation of a valence electron from sodium to chlorine. The two other answer combinations have opposite perspectives on whether carbon dioxide has low melting and boiling points but the commonality of using an empirical fact (carbon dioxide is a gas at room temperature) rather than a particulate-level explanation as a reason. Two alternative conceptions were found among first-timers only, that sodium chloride produces free electrons when dissolved in water and that calcium fluoride is made up of covalent molecules. Finally, the alternative conception that sodium chloride is made up of covalent molecules involving a shared pair of electrons is prevalent for repeaters and first-timers, but the students with prep-chem were not, as a group, quite as attracted to this idea.

Table 5 Alternative conceptions about Chemical Bonding held by all three groups of General Chemistry I students (N = 679)
Alternative Conception Choice combination A: With Prep-Chem (N = 348) B: Repeaters (N = 108) C: First-timers (N = 223) Twice % guessing value
Electrical conductivity of ionic compounds
Calcium fluoride is an ionic compound; it has free electrons that enable it to conduct electricity. Item 8 [A3] 42.5 31.5 33.2 25
Structure of sodium chloride
After donating its valence electron to the chloride, the sodium ions form a molecule with the chloride ion. Item 6 [A2] 39.1 39.8 33.2 25
Intermolecular and intramolecular forces
Carbon dioxide has low melting and boiling points because is a gas at room temperature. Item 7 [A4] 29.9 38.9 41.3 25
Carbon dioxide doesn’t have low melting and boiling points because is a gas at room temperature. Item 7 [B4] 28.5 27.7 34.5 25

Table 6 Alternative conceptions about Chemical Bonding held by at least one group of General Chemistry I students (N = 679)a
Alternative Conception Choice combination A: With Prep-Chem (N = 348) B: Repeaters (N = 108) C: First-timers (N = 223) Twice % guessing value
a A bold number is used to indicate being above the cutoff in the last column.
Electrical conductivity of ionic compounds
NaCl produces free electrons when dissolved in water, but not when is solid. Item 9 [A3] 24.7 19.4 26.0 25
Calcium fluoride consists of covalent molecules. Item 8 [B4] 18.9 22.2 30.0 25
Structure of sodium chloride
The sodium atom shares a pair of electrons with the chlorine atom to form a simple molecule. Item 6 [A1] 21.6 32.4 33.6 25

Since we are interested specifically in the role Preparatory Chemistry is supposed to play in helping students understand concepts from secondary school chemistry before they enter General Chemistry I, we focus on the two cases from the descriptive analysis in which the cutoff indicated an alternative conception for the other two groups, but not for the students with prep-chem. The question is whether the percentage of students with the alternative conception is significantly different among the three groups, which can be addressed with a chi-square test. For the alternative conception that heat energy is absorbed by boiling water and released as bubbles, the percentages were found to be significantly different (χ2 (2, N = 679) = 7.25, p = 0.03), with prep-chem significantly lower. For the alternative conception that sodium chloride is made up of covalent molecules involving a shared pair of electrons, again the percentages were found to be significantly different (χ2 (2, N = 679) = 11.00, p = 0.003) with prep-chem significantly lower.

While these statistical tests provide some evidence that Preparatory Chemistry students lack two alternative ideas that the other two groups of students still have, when the curriculum for Preparatory Chemistry is examined, it is not at all clear by what mechanism these two particular alternative conceptions would have been corrected while the other alternative conceptions persisted. As indicated by all three groups' responses to the Diagnostic Instrument, the dismaying news is that a high percentage of all students entering General Chemistry I in Spring 2010 had not yet understood fundamental ideas of the particle theory, even though one group of them took a course that, on the surface, contained exactly the right level of content knowledge for them to be successful on this instrument.

Conclusions

In general, all three groups of students have similar alternative conceptions, which have also been reported in previous studies (Johnson, 1998; Harrison and Treagust, 2002; Mulford and Robinson, 2002; Othman et al., 2008). This is particularly true for the distinction between an atom and an element, the role of stirring in dissolution, the nature of the ionic bond in sodium chloride, and the mechanism by which a molten ionic compound conducts electricity, in which most of the students in each group responded with alternative conceptions.

Two tier multiple-choice instruments like the one used in this study are good formative assessment tools. The use of a second tier (reasoning) provides instructors with more information about student conceptual understanding. The next step, once alternative conceptions have been uncovered, is to determine whether instructional strategies can improve student understanding. These types of assessments then become useful as evaluative resources to examine the effectiveness of instruction in addressing student alternative conceptions.

In terms of the role of Preparatory Chemistry, the ANCOVA results, although statistically significant, showed that the course effect on the overall Diagnostic Instrument score is small. In addition, we were able to find only two alternative conceptions that were less attractive to students with prep-chem as compared to the other two groups. Therefore, it is difficult to conclude that exposure to Preparatory Chemistry helped students' understanding of the particulate nature of matter and chemical bonding in a substantive way. The results of this assessment must be used to rethink the role of that course. If the course intention remains to help students to understand and apply the particulate nature of matter as described in the syllabus, specifically targeting the alternative conceptions found in this study can help.

A contribution of this study has been the implementation of a cutoff derived from a plausible guessing frequency to identify students' alternative conceptions. An approximate “guessing percentage” for each item was calculated so that response patterns that were the result of guessing alone could be reduced to the level of noise. Using twice the approximate guessing value as a cutoff, we were able to identify very common alternative conceptions in the study population.

One of the limitations of this study includes using a low stakes exam to evaluate students' alternative conceptions. Although the Diagnostic Instrument was given under similar conditions as the regular course exams, students knew that it was a low stakes exam. This can influence the performance of the students, reducing the impetus to think hard about subtle differences among response options. Another limitation is that the Diagnostic Instrument was not given again to the students at a later point in the term. Therefore, we do not have any data about whether the students were able to overcome or change the alternative conceptions described in this paper. This next step will be particularly important in the future, to determine to what degree the General Chemistry I course instructors are able to use these results to inform their teaching.

Using this instrument to try to understand the role of Preparatory Chemistry has left us with some outstanding questions for future research. We did identify specific answer combinations that were prevalent in all three populations, but, in the absence of interviews or open response items, we feel unsatisfied about the robustness of these alternative conceptions. Would students have responded in the same way if the questions were phrased slightly differently, or if they were asked to explain their thinking? Constrained by a very large chemistry course-taking population and responsible to instructors who value having information about all students, we do need a multiple-choice approach. While the two-tier Diagnostic Instrument has a significant advantage over a single-item test, we believe there is room for still more refinement. The number of items related to each specific concept within the larger topics of Particulate Nature of Matter and Chemical Bonding limited our understanding with regard to the students' understanding of the specifics. Having multiple items designed to probe for the same misconception would provide a greater certainty that students are answering as they truly think, and are not overly influenced by a small detail of question wording or a momentary distraction. What this study has highlighted for us is the critical importance of having even shorter, more-tightly-focused instruments that help instructors probe particular alternative conceptions at the appropriate time in the curriculum.

The Diagnostic Instrument was created to assess students' understanding about two broad concepts, particulate nature of matter and chemical bonding. It was also intended to identify whether students' understanding about one of the concepts influences their understanding of the other concept. Results from the CFA in this case did not support a two-factor structure, so we were unable to use the instrument for that purpose within this study, and we look forward to others' work on this topic. Regardless, the two-tier Diagnostic Instrument provided useful information related to students' conceptions about the particulate nature of matter and chemical bonding that poses challenges for the courses investigated in this study.

Acknowledgements

The authors thank Sachel Villafañe for her involvement and support in data analysis and writing, and the course instructors for General Chemistry I for being willing to learn about their incoming students' ideas about chemistry.

References

  1. Brown T. A., (2006), Confirmatory factor analysis for applied research, (1st ed.), New York: Guilford Press.
  2. Bunce D. M. and Hutchinson K. D., (1993), The use of the GALT (Group Assessment of Logical Thinking) as a predictor of academic success in college chemistry, J. Chem. Educ., 70, 183–187.
  3. Examinations Institute, (2011), American Chemical Society Division of Chemical Education, Full Year General Chemistry, University of Iowa: Ames, IA..
  4. Costu B., (2008), Big bubbles in boiling liquids: students' view, Chem. Educ. Res. Pract., 9, 219–224.
  5. Crocker L. and Algina J., (2006), Introduction to classical and modern test theory, Wadsworth Pub. Co.
  6. Ebenezer J. V. and Gaskell P. J., (1995), Relational conceptual change in solution chemistry, Sci. Educ., 79(1), 1–17.
  7. Garcia C. A., (2010), Tracking chemistry self-efficacy and achievement in a preparatory chemistry course, Unpublished dissertation.
  8. Harrison A. G. and Treagust D. F., (2002), The particulate nature of matter: Challenges in understanding the submicroscopic world. In J. K. Gilbert, O. De Jong, R. Justi, D. F. Treagust and J. H. Van Driel (ed.), Chemical education: Towards a research-based practice, (pp. 189–212), Dordrecht: Kluwer Academic Publishers.
  9. Hogarty K. Y., Hine C. V., Kromrey J. D., Ferron J. M. and Mumford K. R., (2005), The quality of factor solutions in exploratory factor analysis: The influence of sample size, communality, and overdetermination, Educ. Psychol. Meas., 65(2), 202–226.
  10. Hu L. and Bentler P. M., (1999), Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives, Struct. Equ. Modeling: A Multidisciplinary Journal, 6(1), 1–55.
  11. Jiang B., Xu X., Garcia C. A. and Lewis J. E., (2010), Comparing two tests of formal reasoning in a college chemistry context, J. Chem. Educ., 87(12), 1430–1437.
  12. Johnson P., (1998), Progression in children's understanding of ‘basic’ particle theory: longitudinal study, Int. J. Sci. Educ., 20, 393–412.
  13. Johnson P., (2005), The development of children's concept of a substance: A longitudinal study of interaction between curriculum and learning, Res. Sci. Educ., 35(1), 41–61.
  14. Lewis S. E. and Lewis J. E., (2005), Departing from lectures: An evaluation of a peer-led guided inquiry alternative, J. Chem. Educ., 82(1), 135–139.
  15. Lewis S. E. and Lewis J. E., (2007), Predicting at-risk students in general chemistry: comparing formal thought to a general achievement measure, Chem. Educ. Res. Pract., 8(1), 32–51.
  16. Lewis S. E. and Lewis J. E., (2008), Seeking effectiveness and equity in a large college chemistry course: An HLM investigation of peer-led guided inquiry, J. Res. Sci. Teach., 45(7), 794–811.
  17. MacCallum R. C., Widaman K. F., Hing S. and Zhang S., (1999), Sample size in factor analysis, Psychol. Methods, 4(1), 84–99.
  18. McFate C. and Olmsted III J., (1999), Assessing student preparation through placement tests, J. Chem. Educ., 76, 562–565.
  19. Mulford D. R. and Robinson W. R., (2002), An inventory for alternate conceptions among first-semester general chemistry students, J. Chem. Educ., 79(6), 739–744.
  20. Murphy K. R. and Davidshofer C. O., (2005), Psychological testing: Principles and applications, Upper Saddle River, NJ: Prentice Hall.
  21. Nyachwaya J. M., Mohamed A.-R., Roehrig G. H., Wood N. B., Kern A. L. and Schneider J. L., (2011), The development of an open-ended drawing tool: an alternative diagnostic tool for assessing students' understanding of the particulate nature of matter, Chem. Educ. Res. Pract., 12(2), 121–132.
  22. Othman J., Treagust D. F. and Chandrasegaran A. L., (2008), An investigation into the relationship between students' conceptions of the particulate nature of matter and their understanding of chemical bonding, Int. J. Sci. Educ., 30(11), 1531–1550.
  23. Potgieter M., Ackermann M. and Fletcher L., (2010), Inaccuracy of self-evaluation as additional variable for prediction of students at risk of failing first-year chemistry, Chem. Educ. Res. Pract., 11, 17–24.
  24. Russell A. A., (1994), A rationally designed general chemistry diagnostic test, J. Chem. Educ., 71, 314–317.
  25. Salta K. and Tzougraki C., (2011), Conceptual Versus Algorithmic Problem-solving: Focusing on Problems Dealing with Conservation of Matter in Chemistry, Res. Sci. Educ., 41, 587–609.
  26. Spencer H. E., (1996), Mathematical SAT test scores and college chemistry grades, J. Chem. Educ., 73, 1150–1153.
  27. Stevens J. P., (1999), Intermediate statistics: a modern approach, Mahwah, NJ: Lawrence Erlbaum Associates.
  28. Taber K. S., (2001), Building the structural concepts of chemistry: Some considerations from educational research, Chem. Educ. Res. Pract., 2(2), 123–158.
  29. Tai R. H., Sadler P. M. and Loehr J. F., (2005), Factor influencing success in introductory college chemistry, J. Res. Sci. Teach., 42, 987–1012.
  30. Teichert M. A., Tien L.T., Anthony S. and Rickey D., (2008), Effects of context on students' molecular-level ideas, Int. J. Sci. Educ., 30(8), 1095–1114.
  31. Thompson B., (2003), Score reliability: Contemporary thinking on reliability issues, Thousand Oaks, CA: Sage Publication, Inc.
  32. Treagust D. F., (1986), Evaluating students' misconceptions by means of diagnostic multiple-choice items, Res. Sci. Educ., 16, 199–207.
  33. Treagust D. F., (1988), Development and use of diagnostic tests to evaluate students' misconceptions in science, Int. J. Sci. Educ., 10, 159–170.
  34. Treagust D. F., Chandrasegaran A. L., Zain A. N. M., Ong E. T., Karpudewan M. and Halim L., (2011), Evaluation of an intervention instructional program to facilitate understanding of basic particle concepts among students enrolled in several levels of study, Chem. Educ. Res. Pract., 12(2), 251–261.
  35. Tro N. J., (2009), Introductory chemistry (3rd ed.). Upper Saddle River, NJ: Pearson.
  36. Tsai C. C. and Chou C., (2002), Diagnosing students' alternative conceptions in science, J. Comput. Assist. Learn., 18, 157–165.
  37. Villafañe S. M., Bailey C. P., Loertscher J., Minderhout V. and Lewis J. E., (2011), Development and analysis of an instrument to assess student understanding of foundational concepts before biochemistry coursework, Biochem. Mol. Biol. Educ., 39(2), 102–109.
  38. Voska K. W. and Heikkinen H. W., (2000), Identification and analysis of student conceptions used to solve chemical equilibrium problems, J. Res. Sci. Teach., 37(2), 160–176.
  39. Wagner E. P., Sasser H. and DiBiase W.J., (2002), Predicting students at risk in general chemistry using pre-semester assessments and demographic information, J. Chem. Educ., 79, 749–755.

Footnotes

† Electronic supplementary information (ESI) available. See DOI: 10.1039/c0rp90017f
‡ The two authors contributed equally to this work.
This journal is © The Royal Society of Chemistry 2012