View PDF VersionPrevious ArticleNext Article
 
DOI: 10.1039/C4RP00094C (Paper) Chem. Educ. Res. Pract., 2014, 15, 620-627

Pre-assessment and peer tutoring as measures to improve performance in gateway general chemistry classes

R. J. Allenbaugh * and K. M. Herrera
Department of Chemistry, Murray State University, 1201 Jesse D. Jones Hall, Murray, KY 42071, USA. E-mail: rallenbaugh@murraystate.edu

Received 1st May 2014 , Accepted 15th June 2014

First published on 26th June 2014

Abstract

Determining student readiness for gateway chemistry courses and providing underprepared students effective remediation are important as student bodies are growing increasingly diverse in their pre-college preparation. The effectiveness of the ACT Mathematics Test and the Whimbey Analytical Skills Inventory (WASI) in predicting student success in first semester general chemistry at Murray State University was analysed in conjunction with a pilot remediation program focusing on problem solving skills. The WASI assessment was as effective as the ACT Mathematics Test in predicting student success and has the advantages of being a shorter, more readily available assessment that can be given at any time; however the PLA program was not effective in improving student outcomes in general chemistry.

Introduction

Gateway classes are introductory, often freshman, classes for which successful completion is required for a student to move forward in his/her major (Coley, 1973; Pickering, 1975; Ozsogomonyan and Loftus, 1979; McFate and Olmsted, 1999; Mason and Verdel, 2001; Hutchison and Atwood, 2002; Swarat et al., 2004; Jones and Gellene, 2005; Lewis et al., 2009). The focus on introductory courses as portals to academic achievement sometimes overlooks barriers to these gateways including prerequisites. The Chemistry Department at Murray State University (MSU) has recently finished a project examining how student preparation for and remediation during gateway chemistry courses impacts success. This study has two parts: (1) analysis of the effectiveness of pre-assessments in predicting student success and (2) analysis of a critical thinking and mathematical skills tutoring program.

Universities face a growing proportion of students who have graduated from high school, but are unprepared to meet the demands of college generally and gateway courses specifically. In the United States, only 26% of ACT-tested high school students met all ACT College Readiness Benchmarks in 2013 (ACT, 2013). Gateway course faculties are increasingly charged with filling this preparation gap by their universities. Superficially this can be achieved by (1) identifying the deficient students and their areas of difficulty, and (2) instructing students in those areas, preferably concurrently with their usual coursework so as to prevent an increase in time-to-graduation.

The Department of Chemistry at MSU was presented with just such an assignment upon the development of a university-wide committee is tasked with reducing D, E, and W outcomes in gateway classes. (At MSU, the “E” mark is a failing grade, which is equivalent to the “F” used at most institutions, while students receive a grade of “W” if they withdraw from a class after the first week of the semester.) The authors openly acknowledge that this is not the ideal situation for developing a remediation program. The funding provided for remediation was non-reoccurring and the lead-time for implementing the program was short, approximately two months. The situation is, however, a typical one, especially for regional institutions that are working under reduced state funding.

The authors of this study had three goals. The first goal was to find a successful pre-assessment for general chemistry that was rapid, simple to administer, and more readily available that the ACT/SAT. MSU is in the process of expanding its enrollment and is admitting more non-traditional, veteran, community college transfer, and international students. These students frequently have never taken the ACT/SAT or have results so out-of-date that they no longer reflect the students' actual knowledge. For these students a simple pre-assessment that can be taken and scored during the course of a standard one-hour advising session is needed. Whimbey Analytical Skills Inventory, WASI, (Whimbey et al., 2013) was selected as this secondary assessment to be compared to the students' scores on the mathematics portion of the ACT examination, Math ACT. Further details on the WASI are provided in the section on Pre-Assessment Information.

The second goal of this study was to determine if a rapid and intensive remediation would be effective in improving the grades of underprepared students. If remediation was effective, these study results would be utilized as a rationale to increase funding for the program. If remediation was ineffective, then the argument for institutional support beyond that for superficial programs could be made.

Having acknowledged that a preparedness gap exists, and that it is unlikely to be remedied at the high school level in the foreseeable future, students, faculty, and administrators are in need of solutions to help students: the third goal of this study. From the perspectives of all parties, simple solutions that can be quickly implemented are ideal. In the absence of literature documenting that rapid, short-term projects are unsuccessful the administrative push will be to continue these sorts of projects. The authors hope to demonstrate that although such efforts are well-intentioned and can be implemented quickly, they are not always an effective method of helping students and, further, that the fault may not be in the rapid implementation of these programs, but rather in the student commitment to the remediation process.

Pre-assessment information

Currently there is no ACT Benchmark for Chemistry. MSU, like many schools in the United States, uses a minimum ACT Mathematics score as one option for meeting the prerequisite for the gateway chemistry course, General College Chemistry, (CHE 201). Alternatively, students may enter the class once they have obtained a C or above in College Algebra (MAT 140) or with a Math SAT score of at least 550. This multi-option prerequisite is designed to insure that students have the basic mathematical and problem solving skills to do well in CHE 201. MSU currently uses a Math ACT score of 23 as the cutoff to enter CHE 201, a midrange score in comparison to the other public universities in Kentucky.† This study examined the Math ACT's effectiveness as a predictor of student outcomes at MSU because not all literature on its use is positive. While there have been several studies on the use of ACT or SAT scores both alone and in combination with some form of “in-house” assessment as a predictor of success in general chemistry, (Martin, 1942; Pickering, 1975; Ozsogomonyan and Loftus, 1979; McFate and Olmsted, 1999) other studies have questioned the effectiveness of using assessments including the ACT and SAT for placement (Coley, 1973; Wagner and Sasser, 2002).

The use of the mathematics portion of the ACT, rather than the science portion, is supported by the literature. McFate and Olmsted (1999) successfully showed that the skills required for success in general chemistry were not chemistry specific knowledge in a study of the California State University assessment examination. Their study determined that while chemical information was useful, questions involving mathematical facility were better able to discriminate between students likely to be successful (as measured by a course grade of C or better) in general chemistry and those likely to be unsuccessful. Wagner and Sasser (2002) similarly found mathematics to be a stronger predictor of success in general chemistry than chemistry background. More recently, Leopold and Edgar (2008) demonstrated that students with higher mathematical fluency tended to perform better, as measured by course grade, in second semester general chemistry, and Cooper and Pearson (2012) have effectively shown that a diagnostic algebra test can help predict the likelihood a student is “at risk” for earning a low grade in the course.

Like the Math ACT, the Whimbey Analytical Skills Inventory pre-assessment (Pre-WASI) is not a chemistry-specific assessment. First introduced by Whimbey (1985), it is now readily available (Whimbey et al., 2013) and has been evaluated as an entrance assessment at the college level and validated in comparison to other assessments, including the Cornell Critical Thinking Test and the New Jersey Test of Reasoning for use a college-level placement test (Morante and Ulesky, 1984). The 38 question assessment is a combination of multiple choice and single-word short-answer questions that, although untimed, takes approximately 40–50 minutes to complete. Mathematical calculations do not require a calculator. For example, one question requires calculating a simple ratio to determine how far a train travels in the time a car has travelled 60 miles. Other questions ask the test-taker to complete analogies or to determine a pattern to fill in the next value in a series. The WASI post-assessment (Post-WASI), also used in this study, follows the same format as the pre-assessment.

Research methods

PLA program structure and organization

As will be explained in detail in the section entitled “Project Impetus,” students with minimum mathematics skills performed very poorly in CHE 201. Based on the preliminary results discussed in that section, the remediation program analyzed here focused on building “at risk” students' critical thinking and mathematical skills. The importance of peer-interaction in learning (Hockings et al., 2008) led to an emphasis on helping tutoring participants help each other. The tutoring sessions utilized the “Whimbey Pair” method (Whimbey et al., 2013) in which one student solves the problem while verbally explaining every step of his/her thought process, and the other student aids the problem solver by checking the logic of each step and asking for clarification where necessary. These roles are frequently interchanged. Because students with problem-solving difficulties could be expected to have difficulty in using the Whimbey method, especially in the early stages of tutoring, an upperclassman acted as a peer-tutor for all sessions, answering questions and modeling the out-loud critical thinking process. In recognition of the important contribution of the peer tutor in facilitating the tutoring sessions, the remediation program was termed the Peer Learning Assistant (PLA) program and the peer tutor was referred to as a PLA.

The PLA program involved 10 students identified as having difficulty in problem solving and/or analytical thinking using the Pre-WASI assessment. All students agreeing to participate in the study took the Pre-WASI under the supervision of the PLA, and their scores were then ranked. The authors initially planned to randomly select students from the bottom quartile for tutoring; however, students scoring 19 (out of 38) on the Pre-WASI were spread between the third and fourth quartiles. Tutoring participants were therefore randomly selected from all students scoring 19 or less on the Pre-WASI. This corresponds to the lowest 30% of scores on the assessment. The randomly selected students were divided into two groups of five students based on their schedules. Each group met twice each week, and each tutoring session lasted 1.5 hours.

The format of the tutoring program was to introduce a general area of analytical thinking during the first session of the week. This was followed by a chemistry-specific application of the skill(s) learned in the week's first session during the second session. For example, week two of tutoring focused on solving problems with relationships. Students were encouraged to read problem solving methods section in the provided text, (Whimbey et al., 2013) and during the first session of the week, students used the Whimbey Pair method to work problems that involved relationships and/or proportions, but not chemistry. Both the faculty mentor and the PLA were present, and aided students in the Whimbey Pair method, first by modeling the process and then by working with the students as both the problem solver and the listener. In the second tutoring session of the week, students used the skills they had learned in determining numerical relationships in word problems to find stoichiometric relationships when balancing equations. Again, the Whimbey Pair method was utilized. Each week the “activity” portion of tutoring lasted for approximately the first hour of the tutoring session. In the last half-hour, students were encouraged to bring questions from homework, lecture, and/or laboratory, which PLA would help facilitate them working through.

The selection of the PLA was made from outstanding upperclassmen who had previously been laboratory teaching assistants for CHE 201. Upon selection, the PLA and faculty mentor discussed the Whimbey Pair method and techniques for encouraging students to work through problems themselves. Each week the PLA and faculty mentor met to discuss the goals and exercises for each week's tutoring sessions. To insure that the PLA was familiar with the everyday details of the CHE 201 class under study, she attended lectures and took notes which were made available to students as examples.

Data collection methods and statistical analyses

Comparisons were performed using t-test values (t) based on average scores for each assessment. Pre- and Post-WASI scores were compared using a dependent t-test value, while all other values were compared via independent t-test. Pearson's correlation values were determined for all pre-assessments with each of the result measures (American Chemical Society Exam score, grade, and outcome). Strictly speaking, grade and outcome should not be correlated using the Pearson's statistic because grade and outcome data is ranked; however, Pearson's correlations are frequently used for this purpose in the literature, and the values are therefore provided for easy comparison. Spearman's rho values (ρ), used to determine correlation values for ranked data, are a more correct method of analyzing correlation and are discussed in the text. P-values (p) are compared to the 0.01, 0.05, and 0.10 significance levels. Effect sizes were determined from Cohen's d (d) with 0.2, 0.5, and 0.8 being considered small, medium, and large effect sizes, respectively. Because not all students completed all assessments, and because certain assessments do not include all outcomes, the number of data values (N) varies from comparison to comparison as do the degrees of freedom (df). At MSU, grade point average (gpa) is calculated on a 4.0 scale corresponding to A–E grades. Students assigned a grade of audit (AU), withdrawal (W) or incomplete (I) are not included in gpa calculation. For determination of outcome, AU and W grades are included by assigning each AU or I grade a value of 1 and each W grade a value of 0.

The selection of the outcome values for AU and W grades is based on the general circumstances of MSU students who earn these marks. At many universities and in other courses at MSU, students audit due to an interest in material unrelated to their degrees or to take more hours during the semester. In these cases, students normally enter the course having already signed up to audit or audit during the first weeks of the semester. CHE 201 students generally audit in the 12th or 13th week of the semester, just prior to the audit deadline, due to poor performance. Thus, a grade of AU is treated like a D grade because both demonstrate the student continues with the course material for the length of the semester, but he/she will need to repeat CHE 201 to successfully go on to more advanced work. W and E grades are treated similarly because both imply that the student has completely failed to keep up with the coursework.

Five different cohorts of student results were analyzed: (1) “Historical”, all students enrolled in all sections of CHE 201 in the fall and spring semesters during the two years prior to this study (N = 605); (2) “Test Class”, CHE 201 students from one lecture section of CHE 201 that participated in this study (N = 56); (3) “Tutoring”, the group of students in the bottom 30% of scorers on the initial assessment who were randomly selected for participation in the tutoring portion of the study (N = 10); (4) “Control”, the remaining students in the bottom 30% of scorers on the initial assessment who were not selected for tutoring (N = 7); and (5) “Proficient”, the top 70% of scorers on the initial study assessment (N = 39). Control, Tutoring, and Proficient are subgroups of Test Class, but the Test Class is not a subgroup of Historical.

Data collection was carried out with the approval and oversight of the MSU Institutional Review Board (IRB), and all participants signed informed consent statements regarding the project. At the beginning of this study, five students enrolled in the CHE 201 class being analyzed were under 18 years of age. IRB requirements prevented these students from signing informed consent documents. For this reason, they were not included in this study, and they are not a part of Test Class.

Course and participant information

CHE 201 is taught as a combined lecture-laboratory course. Laboratory section consist of a maximum of 24 students, and two to three laboratory sections are combined for the lecture portion of the class. Normally, nine to ten sections of general chemistry are taught in the fall semester (three sections per faculty member) and four sections are taught in the spring (two per faculty member). The professors are, generally, the same for the period of historical analysis (the fall and spring semesters for a period of two years prior to this study) and the study period. All students are given the standardized American Chemical Society (ACS) First Semester General Chemistry form 2009 exam at the end of the semester. Students that participated in the Pre- and Post-WASI testing during the study period received extra credit for participation in the study. Students were also able to earn the same amount of extra credit for participation in enrichment activities (e.g. attending seminars) instead of participation in the study. In only one case did the extra credit change the student's letter grade. For purposes of this study, the gpa values are determined without extra credit for study participation.

CHE 201 is designed as an introductory chemistry course for science majors and students seeking advanced degrees in health-related fields (medicine, pharmacy, veterinary). As a representative example, the nine sections of the course during the study period were analyzed. Test Class comprises students from three of these sections. In all nine sections of the course, 231 students were enrolled. Nearly half the students had freshman status (48%), while 30% were sophomores, 15% were juniors, and 7% were senior or post-baccalaureate students. These students represented 45 declared majors, with <1% of the students undeclared. Within many departments at MSU there are separate majors based on future career tracks which at other institutions might be considered a single major. For example, in the Department of Chemistry there are eight majors including the ACS certified B.S. in Chemistry and majors for each pre-professional track (e.g. Chemistry/pre-pharmacy, Chemistry/pre-medicine). For ease of understanding, the various majors/tracks have been grouped into the more traditional categories. Agriculture Science, Exercise Science, Nursing, and Occupational Health and Safety majors accounted for 9.9% of enrollment. Animal Technology (which includes pre-veterinary and pre-veterinary technician programs) accounted for 18.6% of enrollment. Engineering and Physics related majors accounted for 8.7% of enrollment. Biology majors accounted for 38.1% of enrollment, Chemistry majors 15.2% of enrollment, and Mathematics majors <1% of enrollment. The remaining 8.7% of enrollment was made up of majors not associated with science, engineering, or technology.

Results

Project impetus

In light of McFate and Olmsted's (1999) research into the importance of critical thinking and mathematic skills, the Chemistry Department at MSU first examined the CHE 201 grades for students meeting only the very minimum mathematics skill level prerequisite when entering 201. All students in all sections of CHE 201 during the fall semester of the previous year were analyzed to determine which prerequisite option had allowed them to enter CHE 201. While some students met multiple prerequisite options, most entered by either (1) having a Math ACT score of 23 or above or (2) meeting the minimum requirement by taking College Algebra, MAT 140, or its equivalent during a previous semester. Forty eight students fell into this latter category and had MAT 140 as their highest mathematics class prior to CHE 201. Fig. 1 shows the CHE 201 grades earned by these students based on their MAT 140 or transferred equivalent grades. Only 21% of these students passed the course, and none earned a grade above “D”.
image file: c4rp00094c-f1.tif
Fig. 1 Grades earned by students in the fall prior to this study entering CHE 201 with Math ACT < 23 and College Algebra (MAT 140) as their highest mathematics course as a function of their grade in MAT 140 or its equivalent at another university. MAT 140 meets a prerequisite for CHE 201 for students who have Math ACT scores < 23.

MAT 140 focuses on teaching students algebraic manipulation, setting up and solving equations in several forms, and graphing. While they become proficient in a large variety of algebraic techniques, students most frequently solve for a variable having already been given the equation and are expected to have learned “pre-algebra” skills for solving word problems and reasoning proportionally using fractions and percentages prior to entering MAT 140. Minimally mathematically qualified student entering CHE 201 can typically solve for a variable such as pressure given the formula PV = nRT, yet will have difficulty determining pressure given volume, temperature, and number of moles of a gas in the context of a word problem, even when all constants and equations are provided, consistent with the grades shown in Fig. 1. In discussion with the Mathematics Department, it was determined that helping improve the students' critical thinking and problem solving skills might be an effective way to combat the poor CHE 201 outcomes mentioned above in future classes.

Comparison of test class and previous semesters

Comparisons were made to the Historical group to determine if the Test Class for this study was a typical one at MSU. For the 546 students in the Historical group the average ACT Math score was 23.8 ± 4.1. Fifty nine students, 9.8% of the Historical group, did not report ACT scores to the university. In the Test Class 5.4% of the students did not report a Math ACT score, and the average for the students reporting a score was 24.8 ± 4.0 (N = 53). The Historical group gpa was 2.2 ± 1.3. The Test Class had an average grade of 2.1 ± 1.3. The Test Class was statistically similar (p > 0.05) to the Historical group both prior to the course (based on Math ACT scores) and on end of semester assessments (based on gpa). Although the one point difference in average Math ACT scores is significant on the p < 0.10 level, the difference is trivial due to the small effect size (d = 0.03). Details of the statistical analyses are given in Table 1.
Table 1 Statistical parameters for first semester general chemistry performance on various assessments. Comparisons are presented in rows wherein the statistical information (N, mean, SD) for each group is presented followed by t-test results
Assessment Groupa N Mean SD Groupa N Mean SD df t p d
a Historical: all students taking CHE 201 during the fall and spring semesters in the two years prior to this study; Test Class: all students participating in this study; Control and Tutoring refer to sub-groups of the Test Class, who scored in the bottom 30% of Pre-WASI scorers and either did not (Control) or did (Tutoring) participate in the tutoring program. b Three students in the test class and 59 students in the historical group did not have reported ACT scores. c Outcome was determined by factoring in additional values to the MSU 4.0 grading scheme of A = 4, B = 3, C = 2, D = 1, E = 0. In the outcome score, students withdrawing from the class (W) are assigned an outcome value of 0, while students auditing the class (AU) or with Incomplete grades (I) are assigned an outcome value of 1. d WASI Improvement compares Pre- and Post-WASI scores for each group.
Math ACTb Test Class 53 24.8 4.0 Historical 546 23.8 4.1 597 1.699 0.09 0.03
Math ACTb Proficient 37 25.7 4.0 Tutoring 10 21.1 3.4 45 3.323 0.002 0.19
Math ACTb Control 7 24.7 1.4 Tutoring 10 21.1 3.4 15 2.630 0.02 0.84
Pre-WASI Proficient 39 25.1 3.2 Tutoring 10 17.1 2.9 47 7.228 <0.0001 0.41
Pre-WASI Control 7 16.4 2.6 Tutoring 10 17.1 2.9 15 −0.489 0.63 0.10
Post-WASI Proficient 23 26.8 5.3 Tutoring 5 20.2 6.4 26 2.441 0.02 0.26
Post-WASI Control 6 20.5 4.9 Tutoring 5 20.2 6.4 9 0.088 0.93 0.03
ACS Proficient 23 46.2 11.2 Tutoring 5 35.0 8.4 26 2.101 0.045 0.21
ACS Control 6 32.5 6.6 Tutoring 5 35.0 8.4 9 −0.556 0.59 0.16
Grade Test Class 35 2.1 1.3 Historical 454 2.2 1.3 487 −0.202 0.84 0.01
Grade Proficient 24 2.5 1.4 Tutoring 5 1.6 0.5 27 1.350 0.20 0.13
Grade Control 6 1.2 0.8 Tutoring 5 1.6 0.5 9 −1.069 0.31 0.25
Outcomec Test Class 56 1.5 1.3 Historical 546 1.8 1.4 600 −1.285 0.20 0.02
Outcomec Proficient 39 1.7 1.5 Tutoring 10 1.10 0.7 47 1.283 0.21 0.07
Outcomec Control 7 1.14 0.7 Tutoring 10 1.10 0.7 15 0.121 0.91 0.02
WASI improvementd Control 6 4.3 4.7 5 2.618 0.06 1.07
Proficient 23 0.8 3.4 22 1.154 0.26 0.24
Tutoring 5 3.4 3.9 4 0.926 0.41 0.41

Comparison of initial assessment results

The average Pre-WASI score was 17.1 ± 2.9 for the Tutoring group and 16.4 ± 2.6 for the Control group. These values are not statistically different (p > 0.05) making the Control group a suitable control for any improvements that might result from the tutoring program. The Proficient group of students, those scoring 20 or more points on the Pre-WASI, averaged 25.1 ± 3.2 on the Pre-WASI assessment. Unsurprisingly, these scores are statistically different from those obtained by the students in the Tutoring group. Math ACT scores were also statistically different (p < 0.05) for the Proficient (25.7 ± 4.0) and Tutoring (21.1 ± 1.4) groups, although the effect size (d = 0.19) is small in this case. The Control group Math ACT average (24.7 ± 1.4) was also statistically different than the Tutoring group with a large effect size (d = 0.84).

Terminal assessment results

There were three terminal assessment measures: (1) Post-WASI scores, (2) ACS standardized exam scores, and (3) class grades. The supervised Post-WASI assessments were given during the last week of class and were not taken by students auditing or who had withdrawn from the class. The Proficient group performed statistically better (p < 0.05) on the Post-WASI than the Tutoring group (average scores 26.8 ± 5.3 and 20.2 ± 6.4, respectively) with a medium effect size (d = 0.41). The improvement in WASI scores was not statistically significant for either group.

The Control group performed surprisingly well on the Post-WASI with an average score of 20.5 ± 4.9. Their improvement was statistically significant at the p < 0.10 level, although not at the p < 0.05 level, and the effect size was large (d = 1.07). Despite the statistically significant improvement in scores, the Control group did not perform statistically better on the Post-WASI than the Tutoring group.

On the second assessment, ACS exam scores, the Tutoring group performed worse (p < 0.05) than the Proficient group, averaging a raw score of 35.0 ± 8.4 (45th percentile) compared to the Proficient group average of 46.2 ± 11.2 (77th percentile) with a medium effect size (d = 0.22). The Control group's average raw score of 32.5 ± 6.6 (39th percentile), was not statistically different than the Tutoring group's average. Overall, Test Class averaged 42.1 ± 11.6 (66th percentile) on the exam.

As was discussed previously, average grades for the Test Class as a whole were not statistically different from the Historical Data. The performance of the Tutoring group (gpa 1.6 ± 0.5) was not statistically different than the Proficient group (gpa 2.5 ± 1.4) or the Control group (gpa 1.2 ± 0.8). In addition to the standard gpa calculation of average grade, an average outcome value was also determined for each group to incorporate AU and W grades. The Tutoring group (avg. outcome value 1.10 ± 0.7) did not perform any differently than the Control group (avg. outcome value 1.14 ± 0.7), and neither are statistically different from the Proficient group (avg. outcome value 1.7 ± 1.5).

Student participation in the tutoring program, as measured by percentage of sessions attended, varied widely. The three students in the tutoring program earning C grades in CHE 201 attended an average of 82.7 ± 16.7% of the sessions. The two students earning D grades attended 78.0% of session on average. The three auditing students attended an average of 54.7 ± 37.8% of sessions. The large standard deviation in this group is due to a student who audited in the fourth week of the semester and did not return to tutoring or class. The two students who withdrew from the class attended an average of 34.0% of sessions, all sessions prior to the decision to withdraw from the class. The seven students attending at least 60% of the sessions had an average gpa of 1.6 ± 0.5. All students attending less than 60% of sessions audited or withdraw, and a gpa for this group cannot be determined. Outcome values of 1.4 ± 0.5 those attending 60% or more of the sessions and 0.3 ± 0.6 for those attending fewer sessions can be determined, a difference that is statistically significant (p < 0.05) with a large effect size (d = 0.87).

Pre-assessment correlation with grades and outcomes. Statistical parameters for the correlation coefficient determination are presented in Table 2. There were two pre-assessments, the Math ACT and the Pre-WASI, which correlated moderately with each other (r = 0.56, N = 53) a result that is statistically significant (p < 0.01). A statistically significant (p < 0.01) correlation value of 0.55 (N = 408) was obtained when evaluating the correlation between Math ACT scores and grade for the Historical group. A similar value is obtained when the outcome value is to the Math ACT scores.
Table 2 Pearson's (r) and Spearman's (ρ) correlation values and statistical parameters
Groupa Assessment Math ACTb ACS Exam Grade Outcomec
N r N r N ρ r N ρ r
a Historical: all students taking CHE 201 during the fall and spring semesters the two years prior to this study; Test Class: all students participating in this study; Control and Tutoring refer to sub-groups of the Test Class, who scored in the bottom 30% of Pre-WASI scorers and either did not (Control) or did (Tutoring) participate in the tutoring program. b Three students in the test class and 59 students in the historical group did not have reported ACT scores. c Outcome was determined by factoring in additional values to the MSU 4.0 grading scheme of A = 4, B = 3, C = 2, D = 1, E = 0. In the Outcome score, students withdrawing from the class (W) are assigned an outcome value of 0, while students auditing the class (AU) or with Incomplete grades (I) are assigned an outcome value of 1. d Spearman's correlations, not the Pearson's statistic, should be utilized for ranked data like grade and outcome; however, because the Pearson's correlation is used in the literature, it is provided here for comparison. e These correlations are significant at the 0.01 level (two-tailed). f These correlation are significant at the 0.05 level (two-tailed).
Historical Math ACTb — — — — 408 0.55e 0.40e 547 0.54e 0.45e
Test Class Math ACTb — — 33 0.69e 34 0.67e 0.64e 53 0.56e 0.57e
Test Class Pre-WASI 51 0.56e 34 0.70e 35 0.54e 0.58e 56 0.28f 0.39f

Both assessments were then correlated with the post-assessment metrics for Test Class. The Pre-WASI score was determined to show a statistically significant (p < 0.01) correlation of 0.54 (N = 51) with grade for the Test Class. This is an improvement over the Math ACT-grade correlation observed for the Historical group; however, it is not higher than the Math ACT-grade correlation observed for the test class (ρ = 0.67, N = 34, p < 0.01). The Pre-WASI correlation to outcome is low for the Test Class at only 0.28, which is statistically significant only at the p < 0.05 level. Math ACT scores are not as highly correlated with outcome as they are with grade, but the correlation (ρ = 0.56, N = 53) is still significant at the 0.01 level. The strongest correlation in this study was determined for Pre-WASI assessment and ACS exam results of Test Class (r = 0.70, N = 34, p < 0.01).

Discussion

This study was carried out over a very limited time frame in a single professor's course, so the effects of the professor and the PLA cannot be completely discounted. The effects of small sample size are especially important in the terminal assessments. Due to course audits and withdrawals, the sample size for the Tutoring group is reduced to N = 5 for some measures. While the limitations of a very small sample can restrict the generalizability of the findings, this small study size also has an advantage. Since all students in the Control and Tutoring groups had the same professor and met in the same period, the classroom instruction for both groups was identical.

The pre-assessments in this study are equally successful as predictors of success in CHE 201, and their results correlate well with each other. The Pre-WASI provides a rapid (40–50 minutes for most students) and readily available assessment that can be given at any time. With outcome correlations on par with that of the Math ACT, the availability of the WASI assessment makes it a highly successful alternative, especially for use with transfer students who may not report ACT results during admission. It's correlation with the results of the ACS examination are especially striking. Although the availability of the WASI is a positive factor, caution should be taken with students who have already taken the WASI as their scores might be inflated upon retaking the assessment.

In comparison to the success of the new pre-assessment, the PLA program was less successful. Preliminary examination of the data shows that neither grades nor student outcomes differ statistically when comparing the Tutoring and Proficient groups. This would normally be considered the hallmark of a successful intervention program; however, comparing results for the Tutoring and Control groups shows that this is clearly not the case. Tutoring did not improve the results in CHE 201 beyond the results of the Control group.

It is difficult to ascertain a cause for this largely unexpected result. The rapid design and implementation of the PLA program is an obvious first target for concern; however, other programs with larger scope have had similar results. A six year study at Texas Tech University (TTU) examined the effects of a similar combination of assessment and remediation in general chemistry (Bentley and Gellene, 2005). In addition to its larger size and duration, the TTU study differed from this one in two main aspects: (1) the control group was composed of students predicted to have scored poorly on the pre-assessment based on historical data rather than students who actually earned low scores, and (2) their study carried out remediation and general chemistry in separate semesters rather than concurrently. The TTU students had more time for improvement, given that remediation and the general chemistry course ran consecutively and not concurrently, yet participation in remediation did not lead to statistically insignificant improvement in gpa (p > 0.05) when comparing remediation participants to a control group. It is therefore, not a result of the rapid design and implementation of the program that caused the lack of success when comparing the Tutoring and Control groups.

These seem counterintuitive in light of successful general chemistry tutoring/remediation programs in recent years (Larson and Middlecamp, 2003; Botch et al., 2006; Hockings et al., 2008). While the successful programs vary in structure, timing, and whether the material is presented online or in person, all programs appear to have an open format. That is to say that all students who wish, both those with good prior preparation and those without, may participate in the programs. This open format may be a key to success in remediation programs.

Both this study and that of Bentley and Gellene (2005) focused exclusively on low performing students. It is likely too much to expect for these students to overcome the preparation gap through a single semester of remediation, regardless of whether it is concurrent with the course or a pre-requisite. A limit to the size of the preparation gap that could be overcome mathematical study exercises was also suggested by a recent study by Scott (2014) demonstrating exercises for improving mathematical skill, conducted as part of a chemistry course, are most effective for students with mid-range mathematics ability, as reflected by B or C grades in their mathematics course. Low performing students, those with D grades, saw almost no improvement in their scores after the exercise. While the participants in this study were high school students, the material covered (molecular mass, mole ratios, and simple gas law calculations) is largely similar to that in the first month of a gateway chemistry course at the college level.

Beyond prior preparation, students' commitment to the remediation process plays a role in their success, as both this study and that of Bentley and Gellene (2005) indicate. The variation in PLA program outcome as a function of program attendance demonstrates that the remediation program was far more effective for students who continued their investment in the process. Having become informed of their relatively poor preparation, in comparison to their classmates, as a result of the pre-assessment process, students must choose to put in the extra effort to close the preparedness gap and remain in the course; however, commitment or motivation on the part of a student is a very difficult factor to quantify despite the work of several groups on the subject (Pickering, 1975; Holme, 1994). Future efforts in increasing student “buy in” to the remediation process may also increase program effectiveness. An open format is likely to be more effective in this area as well, because students who already have some mathematical facility are less likely to audit or withdraw from the course and will therefore be more likely to stay with the remediation program.

Conclusions

The Whimbey Analytical Skills Inventory pre-assessment (Pre-WASI) was as effective as Math ACT scores in predicting successful outcomes in CHE 201, a first semester general chemistry class for science majors. Given the availability and ease of administration, the Pre-WASI is an excellent alternative when ACT results are unavailable or out-of-date. The PLA program, a peer and faculty mentor based program, held concurrently with the general chemistry class did result in the students within the tutoring program performing at a statistically similar level to the students found to be proficient through pre-assessment as measured by their grades in the class. However, the students in the tutoring group performed no better than those in the control group, so the overall effectiveness programs focused solely on low-performing students must be questioned, especially in light of their cost.

Acknowledgements

The authors would like to thank Murray State University, specifically the Office of the Provost, for funding this project. Data for the Historical group was collected and kindly provided by Dr J. Ricky Cox.

Notes and references

  1. ACT, The condition of college & career readiness 2013, http://https://www.act.org/research/policymakers/cccr13/pdf/CCCR13-NationalReadinessRpt.pdf, accessed Feb, 2014.
  2. Bentley A. B. and Gellene G. I., (2005), A six-year study of the effects of a remedial course in the chemistry curriculum, J. Chem. Educ., 82, 125–130.
  3. Botch B., Day R., Viming W., Stewart B., Rath K., Peterfreund A. and Hart D., (2006), Effects on student achievement in general chemistry following participation in an online preparatory course. Chemprep, a voluntary, self-paced, online introduction to chemistry, J. Chem. Educ., 84, 547–553.
  4. Coley N. R., (1973), Prediction of success in general chemistry in a community college, J. Chem. Educ., 50, 613–615.
  5. Cooper C. I. and Pearson P. T., (2012), A genetically optimized predictive system for success in general chemistry using a diagnostic algebra test, J. Sci. Educ. Technol., 21, 197–205.
  6. Hockings S. C., DeAngelis K. J. and Frey R. F., (2008), Peer-led team learning in general chemistry: Implementation and evaluation, J. Chem. Educ., 85, 990–996.
  7. Holme T. A., (1994), Providing motivation for the general chemistry course through early introduction of current research topics, J. Chem. Educ., 71, 919–921.
  8. Hutchison A. R. and Atwood D. A., (2002), Research with first- and second-year undergraduates: a new model for undergraduate inquiry at research universities, J. Chem. Educ., 79, 125–126.
  9. Jones K. B. and Gellene G. I., (2005), Understanding attrition in an introductory chemistry sequence following successful completion of a remedial course, J. Chem. Educ., 82, 1241–1245.
  10. Larson T. and Middlecamp C. H., (2003), A companion course in general chemistry for pre-education students, J. Chem. Educ., 80, 165–170.
  11. Leopold D. G. and Edgar B., (2008), Degree of mathematics fluency and success in second-semester introductory chemistry, J. Chem. Educ., 85, 724–731.
  12. Lewis S. E., Shaw J. L., Heitz J. O. and Webster G. H., (2009), Attitude counts: self-concept and success in general chemistry, J. Chem. Educ., 86, 744–749.
  13. Martin F. D., (1942), A diagnostic and remedial study of failures in freshman chemistry, J. Chem. Educ., 19, 274–277.
  14. Mason D. and Verdel E., (2001), Chemical education research: Gateway to success for at-risk students in a large-group introductory chemistry class, J. Chem. Educ., 78, 252–255.
  15. McFate C. and Olmsted III J., (1999), Assessing student preparation through placement tests, J. Chem. Educ., 76, 562–565.
  16. Morante E. A. and Ulesky A., (1984), Assessment of reasoning abilities, Educ. Leadership, 42, 71–74.
  17. Ozsogomonyan A. and Loftus D., (1979), Predictors of general chemistry grades, J. Chem. Educ., 56, 173–175.
  18. Pickering M., (1975), Helping the high risk freshman chemist, J. Chem. Educ., 52, 512–514.
  19. Scott F. J., (2014), A simulated peer-assessment approach to improving student performance in chemical calculations, Chem. Educ. Res. Pract., DOI: 10.1039/C4RP00078A.
  20. Swarat S., Drane D., Smith H. D., Light G. and Pinto L., (2004), Opening the gateway: increasing student retention in introductory science courses, J. Coll. Sci. Teach., 34, 18–23.
  21. Wagner E. P. and Sasser H., (2002), Predicting students at risk in general chemistry using pre-semester assessments and demographic information, J. Chem. Educ., 79, 749–755.
  22. Whimbey A., (1985), You don't need a special reasoning test to implement and evaluate reasoning training, Educ. Leadership, 43, 37–39.
  23. Whimbey A., Lochhead J. and Narode R., (2013), Problem solving and comprehension, New York, NY: Routledge.

Footnote

† Math ACT scores used as prerequisites for CHE 201 equivalents at other public universities in Kentucky range from 20 to 25, although Western Kentucky University has no Math ACT prerequisite option.
This journal is © The Royal Society of Chemistry 2014