View PDF VersionPrevious ArticleNext Article
 
DOI: 10.1039/D0RP00283F (Paper) Chem. Educ. Res. Pract., 2021, 22, 303-311

Implementation and evaluation of an adaptive online summer preparatory course for general chemistry: Whom does it benefit?†

Scott A. Reid *a, Laura MacBride b, Llanie Nobile a, Adam T. Fiedler a and James R. Gardinier a
aDepartment of Chemistry, Marquette University, Milwaukee, WI 53233, USA. E-mail: scott.reid@marquette.edu
bOffice of Institutional Research, Marquette University, Milwaukee, WI 53233, USA

Received 19th September 2020 , Accepted 28th November 2020

First published on 22nd December 2020

Abstract

General chemistry courses are key gateways for many Science, Technology, Engineering, and Mathematics (STEM) majors. Here, we report on the implementation and evaluation of an adaptive, ALEKS-based online preparatory module (PM) for general chemistry. The module was made available in Summer 2018 at no cost to all students entering any section of general chemistry that fall. Of the 827 students who registered into the PM, 44% fully completed the module, 48% completed part of the module, and 8% did not complete any of the module. Considering students enrolled in first-term general chemistry, we find a marked increase in ACS final exam percentile for students who completed more than 50% of the module. This is suggested to reflect the self-selection of users who were highly motivated and/or likely to succeed in the course, a hypothesis supported by an analysis using an internal diagnostic metric, the predicted first-year quality point average (PQPA). To examine longer term impacts of the PM, we examined performance in subsequent chemistry courses, through second-semester organic, and found that students completing more than 50% of the module outperformed their counterparts across all courses, with the gap largest in first semester general chemistry and narrowing across subsequent courses. Finally, we surveyed students in summer 2020, two years after the PM offering. The survey indicated overall satisfaction with the PM. For students who did not complete the module, primary reasons given were difficulty (29% of respondents) and insufficient time (46%). As the module did not proportionally benefit the target group of underprepared or at-risk students, we suggest tweaks for future implementations.

Introduction

First-year, introductory, general chemistry (GC) courses are gateways for students in many Science, Technology, Engineering, and Mathematics (i.e., STEM) disciplines. Diversifying the STEM workforce could greatly benefit from the development of strategies for ensuring the success of at-risk and underrepresented populations in GC courses. Various measures, including diagnostic exams, tests of formal thought, (Bunce and Hutchinson, 1993; Lewis and Lewis, 2007) SAT scores, (Pickering, 1975; Craney and Armstrong, 1985; Nordstrom, 1990; Spencer, 1996; Lewis and Lewis, 2007) ACT scores, (Carmichael, et al., 1986; Nordstrom, 1990; House, 1995), and high school GPA (Carmichael, et al., 1986) or chemistry grade, (Craney and Armstrong, 1985; Nordstrom, 1990) have been used to identify or section student groups for remediation or intervention in GC. However, the literature on the efficacy of these courses is mixed. Particular insight comes from a six-year Texas Tech study of a remedial course with in-house placement exam, which included some 6000 students (Bentley and Gellene, 2005). Classifying students into groups of: (a) unremediated–underprepared, (b) remediated–underprepared, or (c) prepared, the data showed no statistical difference in performance in first-term general chemistry between these groups at the 95% confidence level, and a modest difference at the 90% confidence level, amounting to roughly ½ of a letter grade. Importantly, it was found that 44% of successfully remediated students (i.e., those completing the remedial course with a grade of C or higher) did not continue into the main course sequence, leading overall to a decrease in the number of students completing first-term general chemistry. A similar trend has been observed in other studies (Hunter, 1976; Pedersen, 1976).

A follow-up to the Texas Tech study involved a survey of students who completed the remedial course, but did not continue farther (Jones and Gellene, 2005). The primary reason identified was a change of major or degree program, with the importance of the remedial course experience being grade dependent. For students receiving an A in the remedial course, ∼10% indicated that the course was a factor in their decision, while for students receiving a C, this percentage increased to ∼50%. The latter group also gave a neutral overall rating to the course.

Studies like these have raised the important point of whether preparatory or remedial courses provide any benefit for at-risk student populations. At worst, such courses may act to discourage students from pursuing STEM degree paths, to the degree that they impact affective domain characteristics such as self-efficacy, motivation, attitude, and interest, which are known contributors to student success (House, 1995; Chemers, et al., 2001; Zusho, et al., 2003; Bauer, 2005; Dalgety and Coll, 2006; Ayan, 2010; Galyon, et al., 2011; Lynch and Trujillo, 2011; Ferrell and Barbera, 2015; Nielsen and Yezierski, 2015; Chan and Bauer, 2016; Willson-Conrad and Kowalske, 2018). For example, recent studies in GC courses have shown a connection between student performance on exams and their self-efficacy beliefs (Willson-Conrad and Kowalske, 2018). The importance of attitude in identifying at-risk students and explaining variation in chemistry achievement has also been demonstrated (Bauer, 2005; Bauer, 2008; Brandriet, et al., 2011; Xu and Lewis, 2011; Chan and Bauer, 2014).

Additional insight into the efficacy of preparatory courses comes from the study of a voluntary, self-paced online summer preparatory course for GC (Botch, et al., 2007). A comparison of non-users vs. users who completed more than 50% of the course showed striking differences in final course grades in the first semester course. The authors identified three different student types: (1) those underprepared and unlikely to succeed without additional help, (2) those likely to succeed but considering themselves underprepared, and (3) those likely to succeed and who make use of all available resources. Although the preparatory course was designed for group 1, the authors concluded that the majority of users came from groups 2 and 3. For these students, the preparatory course proved a useful tool, yet the target audience of weaker or less prepared students did not take advantage of the course in proportionate numbers. Consistent with this finding, a recent study of a voluntary online preparatory course for organic chemistry showed that the academic preparation of students was a significant factor in predicting their participation in the course (Fischer, et al., 2019). A similar finding was reported in the study of peer-led team learning (PLTL) groups, where again voluntary participation was suggested to lead to differential participation by more highly motivated students (Chan and Bauer, 2015).

Studies to date have typically examined the utility of prep courses or modules in governing success in the introductory (first-term) course; however, it is also desirable to examine performance in subsequent chemistry courses, through organic chemistry. As part of a larger effort to develop a successful intervention program at our institution, we report here a study of the effect of a voluntary adaptive summer prep module (PM) based on ALEKS. ALEKS, or Adaptive LEarning in Knowledge Spaces, is an adaptive program developed by the ALEKS Corporation (https://www.aleks.com) which is widely used in introductory chemistry, math, and physics. The module was made available by the vendor at no-cost to all students enrolled in general chemistry courses in Fall 2018, and opened shortly following the close of the summer registration period, roughly 6 weeks prior to the start of the fall term. We first examine correlations between final (ACS) exam performance and PM completion for all students who entered the first-term non-majors general chemistry course (N = 676). We then probe correlations with predicted first semester quality point average (PQPA), an internal metric described in detail below, for all first-year students in the cohort for which PQPA data was available (N = 510). We subsequently analyze, for the full cohort, performance and completion rates in subsequent courses including second-term general chemistry and organic chemistry. Finally, we report the results of a student survey administered two years following the offering of the PM.

Research framework and questions

The importance of prior knowledge and skills as predictors of student performance fits squarely within mainstream learning theories, which recognize that students inherently bring a variety of preconceptions and misconceptions into a class. Learning frameworks based on a constructivist perspective highlight the linking of what is being taught to an extant knowledge base (Scerri, 2003; Taber, 2010; Wink, 2014; Pazicni and Flynn, 2019). One framework, Meaningful Learning, (Ausubel, et al., 1978; Halloun, 1996; Brandriet, et al., 2013; Rootman-le Grange and Blackie, 2018) is particularly relevant here, as it principally emphasizes the connection of new information to pre-existing knowledge. Such frameworks are helpful in explaining why misconceptions that learners bring to chemistry (and math, physics, etc.) courses are often quite impervious to further instruction (Bodner, 1986; Zoller, 1990; Abraham, et al., 1994; Sanger and Greenbowe, 1997; Nicoll, 2001; Pinarbasi, et al., 2009; Villafane, et al., 2011; Erman, 2017). From this perspective, remediating student skills and conceptual deficiencies through pre-course efforts should be valuable.

Thus, the research questions which we sought to address here are as follows: (1) What is the effect of an online adaptive, voluntary summer PM on student performance in first-semester general chemistry? (2) What is the longitudinal effect, including attrition rate, of the prep course on performance across second-semester general chemistry and organic chemistry? (3) What is the efficacy of the PM for at-risk students, as identified by institutional metrics? and (4) For students who did not complete the PM, what factors were most important?

Research design, instruments, and participants

General chemistry course structure

The two-semester general chemistry sequence at our institution (see Table S1 for course key, ESI†) typically contains three weekly components: three 50 min lectures, a 50 min discussion section which features small group problem solving, and a 3 hour laboratory. Additionally, some sections use a flipped approach, where the lectures are moved online and a weekly 75 minute discussion meeting is held (Reid, 2016; Ryan and Reid, 2016). Lecture and discussion attendance are counted as 1% (each) of the final course grade, and typical attendance for each is ∼90%. Lectures or flipped discussions are taught by full time faculty or instructional staff, discussion sections by undergraduate or graduate teaching assistants (TAs), and the laboratory by graduate TAs. The course enrolls some 750 students each fall; in Fall 2018, six sections were offered. In four of these PM completion was counted as 1% extra credit, in the remaining sections, it was counted as 1% course credit. Historical DFW (i.e., D's, F's and withdrawal) rates in the first term course are 10–15%.

ACS standardized exams

Standardized first-term general chemistry exams from the American Chemical Society Exams Institute were given as the final exam across all sections. As three different ACS exams were used (First Term General Chemistry Forms 2005, 2009, and 2015), in our analysis we converted the raw ACS exam scores to percentile scores using the published national norms (ACS Examinations Institute, 2020).

ALEKS and the ALEKS preparatory module

ALEKS, which stands for Assessment and LEarning in Knowledge Spaces,32 is a Web-based, artificially intelligent assessment and learning system that employs adaptive questioning. Students are presented an initial assessment, or “knowledge check” consisting of approximately 20 questions over the range of course topics – the selection of questions is adaptive, so that each student receives a unique set of questions. Upon completion, the topics mastered by the student, as well as those not yet learned, are presented in the form of a pie chart, and students are presented with a set of topics to learn. Each topic requires several consecutive questions to be answered correctly in order to demonstrate mastery. Complete explanations are provided for students needing assistance.

For the summer prep implementation used here, students were only required to learn the complete set of topics to complete the course, no periodic re-checks of knowledge were scheduled. The module consisted of 137 topics spanning basic mathematics and physics to specific chemistry knowledge (Table S2 shows included topics, ESI†). At the close of the final summer registration session for new first-year students, in early July 2018, an email invitation was sent, containing a detailed Quick-start/FAQ sheet (ESI†), to all students enrolled in first-term or second-term (off-semester) general chemistry in Fall semester 2018. The preparatory course was opened for enrollment in mid-July, and kept open until the end of the second week of fall classes (early September). The composition of the GC1-nonM (Table S1, ESI†) sections by major was ∼25% engineering (first- or second-year depending on major), ∼45% health sciences, ∼15% biology, with all other majors comprising ∼15%.

Predicted first-year quality point average

The predicted first-year quality point average (or PQPA) is a metric developed by our Office of Institutional Research. This relatively simple predictor models an accepted student's first-year Quality or Grade Point Average based on their high school GPA, ACT/SAT test scores, and college of entry (students at Marquette are admitted by college). We have recently shown that the PQPA strongly correlates with student success in general chemistry, and developed a cutoff threshold (2.8) for identifying at-risk students (Vyas, et al., 2020). The PQPA values in the dataset here ranged from 2.08 to 4.00 on a 4 point scale.

Longitudinal analysis

As a component of this work, we analyzed student grades not only in first-term general chemistry but also second-term general chemistry and first- and second-term organic chemistry. Here, grades were collected from the on-semester offerings of these courses, and retakes were not included in our analysis. All student data was de-identified prior to analysis.

Student survey

The survey plan and instrument used in this work was reviewed and approved by the Online Survey Review Group at our institution, and the complete instrument is included in the ESI.† Following an initial opt-in, students were asked to complete a set of Likert-style questions regarding their participation in and views of the summer 2018 prep course. A closing set of questions captured demographic information. The survey was sent electronically to 827 students in summer 2020, with 255 students completing the survey (response rate of 31%).

IRB review and FERPA compliance

As the data examined in this study represents normal course data, and analysis of the data was performed with deidentified data – i.e., no names or student ID numbers were recorded into the research dataset, this study (HR-8562) was granted an exemption from IRB review. The study was also fully compliant with FERPA policies and procedures.

Results and discussion

A group of 827 students entered into the summer 2018 online PM. Of these, 44% fully completed the module, 48% completed part of the module, and 8% did not attempt or complete any of the module. In addition to degree of completion, the time spent in the module and associated learning rate, measured in topics completed per h, were examined. Fig. S1 (ESI†) shows module completion % vs. time spent (in minutes). The range of time spent for students completing 100% of the module spans a large range, from ∼300 to more than 5000 minutes. The latter seems excessive, yet reflects a learning rate of nearly 2 topics per h, which is not unusual for our in-semester ALEKS implementations. Table S3 (ESI†) compares the aggregate learning rates for students completing part vs. all of the PM. It is striking that the learning rate of students who completed only part of the module was roughly twice that of students who fully completed the module, while both distributions show significant (positive) skewness and kurtosis.

We examined bivariate correlations of preparatory module completion with ACS final exam score, and partial correlations controlling for time spent in the module. These correlations were examined for all students who completed the course and attempted the PM, Table 1. The ACS final exam percentile displayed a moderate correlation with PM completion, which increased when controlling for time spent in the PM. If only first semester students (i.e., those with a PQPA in the data set) were considered, the correlations were similar. A similar correlation was also found between the ACS final and PQPA, Table 1, while a weaker correlation was observed with initial ALEKS knowledge check.

Table 1 Bivariate and partial Pearson correlations, Fall 2018
Variable 1 Variable 2 N Control Pearson correlation Significance level
ACS final exam percentile Preparatory module completion 676 None 0.51 <0.01
Time spent in preparatory module 0.55 <0.01
510 Students w/PQPA (1st semester) 0.51 <0.01
Time spent in preparatory module 0.57 <0.01
ACS final exam percentile Initial ALEKS knowledge check 676 None 0.35 <0.01
ACS final exam percentile PQPA 510 None 0.58 <0.01
Preparatory module completion PQPA 578 None 0.46 <0.01
Initial ALEKS knowledge check PQPA 578 None 0.32 <0.01

Following the approach employed by Botch and co-workers, (Botch, et al., 2007) we initially binned students into subgroups by module completion as: (1) 50–100% completion, (2) attempted with partial completion less than 49%, (3) did not attempt. However, the differences in mean final exam score of the latter two groups was not statistically significant, and therefore we report here an analysis using only two groups: (1) 50–100% completion, and (2) 0–49% completion. Table 2 shows the relevant means and the results of an independent samples T-test for these groups, compared both for the full cohort, and first semester students for which PQPA data was available. The difference in means is significant at the p < 0.001 level with positive and large effect sizes.

Table 2 Comparison of data for subgroups by PM completion, Fall 2018
Cohort PM completion (%) N Mean ACS final percentile (SD)a p-Value of difference Effect size
a Standard deviation given in parenthesis.
All students ≥50 528 71.5 (25.3) <0.001 1.1
<50 148 43.3 (23.7)
PQPA only ≥50 395 71.4 (25.5) <0.001 1.1
<50 115 43.6 (23.9)

The data in Table 2 reveals a marked difference in ACS final exam percentile for the two subgroups. We suspect that this trend is due to self-selection of users who were likely to succeed in the course, but either lacked confidence in their preparation or were motivated to make use of every available resource (Botch, et al., 2007). This supposition is reinforced by examining PM completion vs final course grade. For students (N = 66) who received a final first-semester general chemistry course grade of D, F or W (withdrawal), only 45% completed more than 50% of the PM, and only 12% fully completed the module. In comparison, of students who received a final course grade of C– or higher (N = 662), 80% completed more than 50% of the PM, and 48% fully completed the module.

Probing further, we performed an independent samples T-test with PQPA as the controlling variable. Setting a cutoff PQPA of 2.80, a cut point recently validated for identifying at-risk students, (Vyas, et al., 2020) a comparison of PM completion for the two subgroups is shown in Table 3. The mean difference in completion was ∼36% (p < 0.001), with a large effect size. If we instead consider students who partially or completely finished the PM (N = 465) vs. those who enrolled into the PM but did not take the initial knowledge check (N = 45), we find a mean difference in PQPA of 0.25 (p < 0.001, effect size of 0.86). This reinforces our supposition that students who were at-risk or least prepared did not take advantage of the PM to the same degree as other students.

Table 3 Comparison of preparatory module completion vs. PQPA group
Group N Mean preparatory module completion (SD)a p-Value of difference Effect size
a Standard deviation given in parenthesis.
PQPA > 2.8 471 67.9 (26.6) <0.001 1.3
PQPA < 2.8 39 32.2 (19.1)

In a subsequent phase of analysis, we examined the longitudinal performance of first year students in the PM through the first two-years of college chemistry (general and organic). For analysis of course grades, we kept the same binning for PM completion, and binned students by PQPA into three distinct sub-bins (PQPA = 2.00–3.00, 3.01–3.50, >3.50) for comparison. For students in each bin, we calculated the mean QPA across four courses; first and second term non-majors general chemistry (GC1-nonM/GC2-nonM; Table S1, ESI†), and first and second term organic chemistry (OC1-nonM/OC2-nonM; Table S1, ESI†). For students retaking the courses, we only analyzed the result of their initial attempt. Fig. 1 shows the complete results, where red is consistently used to label the group completing less than 50% of the prep module, and blue the group completing 50% or more. In each panel, bars indicate the number of students, while markers indicate the mean QPA, shown with one standard deviation error bars. Black arrows mark the y-axis associated with each data set.


image file: d0rp00283f-f1.tif
Fig. 1 Student performance across the non-majors general and organic courses by PM completion (blue, >50%, red, <50%) and PQPA group. Within each panel, the bars indicate the number of students in each prep completion group, while the markers indicate mean course QPA. Black arrows identify the y-axis for each data set.

Fig. 1 shows that across all PQPA groups and all courses, students completing at least 50% of the PM received higher grades; however, the difference between these groups typically narrowed across the course sequence. This trend is less meaningful for the highest PQPA group due to the very small number of students who did not complete the PM. However, in the lowest PQPA group, the difference in QPA between subgroups was roughly 1 unit for GC1-nonM, and narrowed to ∼0.2 unit for the final course in the sequence, OC2-nonM. Consistent with our finding above that less well-prepared students did not take advantage of the prep module to the same degree as other students, the prep module groupings are nearly equal for the lowest PQPA cohort, while for the middle and highest PQPA cohorts the group completing at least 50% of the prep module was dominant.

Recently, Freeman and co-workers examined course data over a 15-year period for students in general and organic chemistry at the University of Washington (Harris, et al., 2020). It was reported that underrepresented students performed more poorly in the first semester general chemistry course, and those obtaining low grades were more likely to drop out of STEM majors. However, those obtaining sufficient grades to move on in the sequence were more likely to continue than their peers. Similarly, we find (Fig. 1) that the differential in course grades between students in the lowest and highest PQPA cohort significantly decreased over the sequence of courses. For example, in GC1-nonM the mean course QPA by PQPA cohort was: 2.14 (lowest); 2.99 (middle); 3.64 (highest). By the final course in the sequence, OC2-nonM, the gap in mean course QPA had significantly narrowed (lowest = 3.11; middle = 3.31; highest = 3.65).

These trends could be influenced by demographics, and by selective attrition. Considering demographics, the lowest PQPA cohort had its highest population of students from the College of Arts and Sciences (68%), while the middle and highest PQPA cohorts were dominated by students from Health Sciences (66% and 74%, respectively), which contain a large population of pre-professional students. However, Arts and Science majors included those which require two years of chemistry (e.g., Biology, Biochemistry, Chemistry), as well as those not requiring chemistry at all (e.g., Psychology), where chemistry is taken as preparation for professional school. Thus, all PQPA cohorts included significant populations of students required to take two years of college chemistry for their degree or professional program of interest.

Considering selective attrition, we examined more carefully the attrition rates between first-term and second-term general chemistry and first- and second-term organic chemistry in each subgroup, and this data is presented in Table 4. While the numbers in some subgroups are small, in each subgroup we generally see a smaller attrition rate for students in the higher PM completion group, which may contribute to the trends observed in Fig. 1. This is particularly pronounced in the lowest PQPA group. The exception to these trends, interestingly, is in the highest PQPA group in general chemistry, where a larger attrition rate is observed in the higher PM completion group. Given the very small number of students who completed less that 50% of the PM, this difference may not be significant. Moreover, as the mean course QPA of students in this group was 3.69, any attrition clearly is due to other factors (e.g., a change of major or degree program, delaying the second-term course until summer).

Table 4 Comparison of attrition rates for general and organic chemistry by subgroup
PQPA group Course sequence PM completion (%) N Attrition rate (%)
High General (non-majors) ≥50 91 30
<50 6 17
Organic (non-majors) ≥50 49 16
<50 3 33
Middle General (non-majors) ≥50 246 23
<50 50 32
Organic (non-majors) ≥50 105 25
<50 21 33
Low General (non-majors) ≥50 56 36
<50 59 44
Organic (non-majors) ≥50 17 22
<50 18 42

To further examine the longitudinal impact of the PM, we examined Pearson correlations between course grades across general and organic chemistry, PM completion, and PQPA. These are presented in Table S4 (ESI†). Perhaps unsurprisingly, the correlations with PM completion and PQPA, which are all significant at the p = 0.01 level, are highest in first-term general chemistry and drop steadily thereafter. However, the drop in correlation with PM completion from first-term general chemistry to second-term organic (∼15%) in noticeably smaller than that with PQPA (37%).

The data presented thus far show that: (1) completion of a summer online prep module is correlated with success in first-term general chemistry and beyond, (2) students taking advantage of the course were primarily those who were already prepared and highly motivated to succeed, with underprepared or at-risk students not taking advantage of the course to the same degree, (3) the differential in course grades between underprepared and well-prepared students significantly decreased over the sequence of courses from first-term general chemistry to second-term organic, while (4) the attrition rate overall was larger for less well-prepared students as measured by PM completion. To better understand these trends, we turn to the results of the student survey.

The survey was administered electronically. An invitation to the survey (instrument provided in the ESI†) was sent via email to 827 students, with a follow-up reminder sent roughly two weeks later. Ultimately, 255 responses were received (31% response rate). Unsurprisingly, the demographics of survey respondents were significantly different from that of the entire set, with 76% of survey respondents female (59% for the entire set) and 64% of respondents indicating that they had fully completed the preparatory module (44% for the entire set). In Fig. 2, we show the mean results from a response to seven questions concerning the preparatory module, with responses on a 5-point Likert scale from 1 (strongly agree) to 5 (strongly disagree). It is noteworthy that respondents were neutral on the questions of whether, looking back, they should have spent more time in the module (Q4) or whether the module was a benefit in subsequent courses (Q7). The logic for including the latter question was that, while the majority of PM topics covered first-semester material (see Table S2 in the ESI†), many of these topics (e.g., dimensional analysis, chemical nomenclature, fundamental reactivity and energy concepts) can be considered foundational to success in chemistry. Thus, the goal of this question was to capture student perceptions of the longitudinal effect of the module, given trends in the data which show disparities between PM completion subgroups that persist across multiple courses.


image file: d0rp00283f-f2.tif
Fig. 2 Survey questions and responses on a 5 point Likert scale, from 1 (strongly agree) to 5 (strongly disagree).

The majority of participants agreed that the module was helpful (Q1) and that they would recommend it to other students (Q6). In our offering of the module, help outside of the ALEKS program (e.g., virtual instructor or TA office hours) was not made available; however, a majority of students agreed that such help would be desirable (Q5).

A separate section of the survey focused on students who had not fully completed the module – here, students were asked to identify the most important factor(s) which hindered their completion of the module. Interestingly, 29% of respondents indicated that the preparatory module was too difficult, while 46% indicated that they did not have sufficient time to complete the module due to other activities (work, etc.). Another 12% of respondents indicated that they either chose not to participate or already considered themselves well prepared for college level chemistry.

In light of these responses, we note that the initial ALEKS knowledge check (topic mastery) of students who ultimately completed more than 50% of the PM was 33%, in comparison with 22% for students who ultimately completed less than 50% of the PM. However, we found that the learning rate of students who completed only part of the module was roughly twice that of students who fully completed the module. Thus, as there were no restrictions on the ordering of topic completion, it is plausible that students who found the module difficult overall were successful in completing some topics but were challenged by others. Moreover, as the time required to complete the module was larger for students who began with lower initial mastery, the time available to them may have been insufficient to complete all the PM. In that vein, we note that the mean completion time for students completing 100% of the PM was roughly 27 hours. As the PM was opened in early July and kept open for roughly 8 weeks, this amounted to roughly 3 hours of work per week.

The survey responses also highlight possible unintended consequences of the PM. While it is designed to ensure that all students are well prepared for the course, students who fail to complete the PM may experience additional anxiety or doubt. Such anxiety may also be felt by students who are time challenged to complete the module due, e.g., to full-time summer employment. In our implementation, we extended the due date for the preparatory module two weeks into the fall term, in order to provide additional support for students. This had the positive benefit of significantly increasing attendance at early semester office hours, which are normally not heavily populated. However, students struggling to complete the module may not have taken advantage of this resource.

In closing, we ask: is an adaptive summer prep module for general chemistry of benefit, and if so, whom does it benefit? Our analysis shows that it does benefit students, but not all students, for it is clear that on balance students who need the module the most utilize it less. Thus, the module may only serve to exacerbate the fault lines which already exist between prepared and underprepared students. Moving forward, we consider several tweaks to augment the utility of the module. First, for students who find the module too difficult, or have little time to complete it, a greater emphasis could be directed towards completing some of the module, as opposed to all. This could be achieved by breaking the module into smaller submodules, with students allowed to choose which if any modules to complete, or by awarding credit for completing some fraction, say ½, of the module, with a smaller bonus given for additional completion. A second tweak could provide scaffolded support in the form of virtual instructor/TA office hours in the summer period. Finally, by extending the due date for the module farther into the term, peer led team learning (PLTL) sessions (Mitchell, et al., 2012; Shields, et al., 2012; Street, et al., 2012; Smith, et al., 2014; Rein and Brookes, 2015; Repice, et al., 2016; Robert, et al., 2016; Wilson and Varma-Nelson, 2016; Williams, et al., 2017; Lewis, 2018; Stanich, et al., 2018) could be held in the early part of the semester for students who did not have the opportunity to complete the module in the summer period.

Limitations

The results of this study provide insight into who is benefited by a voluntary online summer preparatory module for general chemistry. However, significant questions still remain as to why this is the case. While the survey instrument used here, which was of limited scope, did aid in identifying two significant factors (degree of difficulty, time commitment) which may play a role, ultimately a more detailed understanding of the barriers faced by students who are less academically prepared is needed, including a deeper understanding of the role and evolution of affective domain characteristics such as self-efficacy, motivation, and attitude.

Conclusions

We have reported on the implementation and evaluation of a summer adaptive, ALEKS based online prep module for first term general chemistry. The module was made available in summer 2018 at no cost to all students enrolled in general chemistry in the fall semester, and was opened at the close of the summer registration period for new students (July). A total of 827 students enrolled in the course. Of these, 44% fully completed the module, while 48% completed at least part of the module. We find a marked increase in ACS final exam percentile score for students completing more that 50% of the module. However, consistent with prior studies, we suggest that this reflects self-selection by users who were highly motivated and/or likely to succeed in the course. This hypothesis is supported by comparison with an internally derived metric, the predicted first year quality point average, or PQPA. Setting a cutoff PQPA of 2.80, a comparison of PM completion revealed a mean difference of ∼36%.

In order to examine the longitudinal effect of the prep module, we examined course grades in subsequent chemistry courses through second-term organic, and conducted a survey of students two years after the offering of the module. Longitudinally, we find that students completing at least 50% of the module outperformed their peers across all chemistry courses in the sequence. However, the gap did narrow across the sequence of courses. Results of the student survey showed that overall, students were satisfied with the prep module. For students who were not able to complete the module, the primary factors identified were the difficulty of the module and a lack of time. Moving forward, implementation of the PM must include strategies to ensure the participation of, and benefit to, the target population of at-risk or underprepared students.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

We gratefully acknowledge Tim Maruna and McGraw-Hill for providing access to the ALEKS program in summer 2018 at no-cost to our students, and the OIRA staff at Marquette for their assistance in data analysis and advice regarding survey design.

References

  1. Abraham M. R., Williamson V. M. and Westbrook S. L., (1994), A Cross-Age Study of the Understanding of 5 Chemistry Concepts, J. Res. Sci. Teach., 31, 147–165.
  2. ACS Examinations Institute, https://uwm.edu/acs-exams/, accessed March 10, 2020, 2020.
  3. Ausubel D. P., Novak J. D. and Hanesian H., (1978), Educational psychology: a cognitive view, New York: Holt, Rinehart and Winston.
  4. Ayan M. N. R., (2010), Motivational patterns defined by cluster analysis and their relationship with perceived ability and academic achievement, An. Psicol., 26, 348–358.
  5. Bauer C. F., (2005), Beyond “student attitudes”: Chemistry self-concept inventory for assessment of the affective component of student learning, J. Chem. Educ., 82, 1864–1870.
  6. Bauer C. F., (2008), Attitude towards chemistry: A semantic differential instrument for assessing curriculum impacts, J. Chem. Educ., 85, 1440–1445.
  7. Bentley A. B. and Gellene G. I., (2005), A Six-Year Study of the Effects of a Remedial Course in the Chemistry Curriculum, J. Chem. Educ., 82, 125–130.
  8. Bodner G. M., (1986), Constructivism - a Theory of Knowledge, J. Chem. Educ., 63, 873–878.
  9. Botch B., Day R., Vining W., Stewart B., Rath K., Peterfreund A. and Hart D., (2007), Effects on student achievement in general chemistry following participation in an online preparatory course - ChemPrep, a voluntary, self-paced, online introduction to chemistry, J. Chem. Educ., 84, 547–553.
  10. Brandriet A. R., Ward R. M. and Bretz S. L., (2013), Modeling meaningful learning in chemistry using structural equation modeling, Chem. Educ. Res. Pract., 14, 421–430.
  11. Brandriet A. R., Xu X. Y., Bretz S. L. and Lewis J. E., (2011), Diagnosing changes in attitude in first-year college chemistry students with a shortened version of Bauer's semantic differential, Chem. Educ. Res. Pract., 12, 271–278.
  12. Bunce D. M. and Hutchinson K. D., (1993), The Use of the Galt (Group Assessment of Logical Thinking) as a Predictor of Academic-Success in College Chemistry, J. Chem. Educ., 70, 183–187.
  13. Carmichael J. W., Bauer S. J., Sevenair J. P., Hunter J. T. and Gambrell R. L., (1986), Predictors of First-Year Chemistry Grades for Black-Americans, J. Chem. Educ., 63, 333–336.
  14. Chan J. Y. K. and Bauer C. F., (2014), Identifying At-Risk Students in General Chemistry via Cluster Analysis of Affective Characteristics, J. Chem. Educ., 91, 1417–1425.
  15. Chan J. Y. K. and Bauer C. F., (2015), Effect of Peer-Led Team Learning (PLTL) on Student Achievement, Attitude, and Self-Concept in College General Chemistry in Randomized and Quasi Experimental Designs, J. Res. Sci. Teach., 52, 319–346.
  16. Chan J. Y. K. and Bauer C. F., (2016), Learning and studying strategies used by general chemistry students with different affective characteristics, Chem. Educ. Res. Pract., 17, 675–684.
  17. Chemers M. M., Hu L. T. and Garcia B. F., (2001), Academic self-efficacy and first-year college student performance and adjustment, J. Educ. Psychol., 93, 55–64.
  18. Craney C. L. and Armstrong R. W., (1985), Predictors of Grades in General-Chemistry for Allied Health Students, J. Chem. Educ., 62, 127–129.
  19. Dalgety J. and Coll R. K., (2006), Exploring first-year students’ chemistry self-efficacy, Int. J. Sci. Educ., 4, 97–116.
  20. Erman E., (2017), Factors Contributing to Students' Misconceptions in Learning Covalent Bonds, J. Res. Sci. Teach., 54, 520–537.
  21. Ferrell B. and Barbera J., (2015), Analysis of students' self-efficacy, interest, and effort beliefs in general chemistry, Chem. Educ. Res. Pract., 16, 318–337.
  22. Fischer C., Zhou N., Rodriguez F., Warschauer M. and King S., (2019), Improving College Student Success in Organic Chemistry: Impact of an Online Preparatory Course, J. Chem. Educ., 96, 857–864.
  23. Galyon C. E., Blondin C. A., Yaw J. S., Nalls M. L. and Williams R. L., (2011), The relationship of academic self-efficacy to class participation and exam performance, Soc. Psychol. Educ., 15, 233–249.
  24. Halloun I., (1996), Schematic modeling for meaningful learning of physics, J. Res. Sci. Teach., 33, 1019–1041.
  25. Harris R. B., Mack M. R., Bryant J., Theobald E. J. and Freeman S., (2020), Reducing achievement gaps in undergraduate general chemistry could lift underrepresented students into a “hyperpersistent zone”, Sci. Adv., 6, eaaz5687.
  26. House J. D., (1995), Noncognitive Predictors of Achievement in Introductory College Chemistry, Res. High. Educ., 36, 473–490.
  27. Hunter N. W., (1976), Chemistry Prep Course That Seems to Work, J. Chem. Educ., 53, 301–301.
  28. Jones K. B. and Gellene G. I., (2005), Understanding attrition in an introductory chemistry sequence following successful completion of a remedial course, J. Chem. Educ., 82, 1241–1245.
  29. Lewis S. E., (2018), Goal orientations of general chemistry students via the achievement goal framework, Chem. Educ. Res. Pract., 19, 199–212.
  30. Lewis S. E. and Lewis J. E., (2007), Predicting at-risk students in general chemistry: comparing formal thought to a general achievement measure, Chem. Educ. Res. Pract., 8, 32–51.
  31. Lynch D. J. and Trujillo H., (2011), Motivational Beliefs and Learning Strategies in Organic Chemistry, Int. J. Sci. Math. Educ., 9, 1351–1365.
  32. Mitchell Y. D., Ippolito J. and Lewis S. E., (2012), Evaluating Peer-Led Team Learning across the two semester General Chemistry sequence, Chem. Educ. Res. Pract., 13, 378–383.
  33. Nicoll G., (2001), A report of undergraduates' bonding misconceptions, Int. J. Sci. Educ., 23, 707–730.
  34. Nielsen S. E. and Yezierski E., (2015), Exploring the Structure and Function of the Chemistry Self-Concept Inventory with High School Chemistry Students, J. Chem. Educ., 92, 1782–1789.
  35. Nordstrom B. H., (1990), Predicting Performance in Freshman Chemistry, Abstr. Pap. Am. Chem. Soc., 199, 234–Ched.
  36. Pazicni S. and Flynn A. B., (2019), Systems Thinking in Chemistry Education: Theoretical Challenges and Opportunities, J. Chem. Educ., 96, 2752–2763.
  37. Pedersen L., (1976), Lower Level Freshman Chemistry - How to Choose Audience, J. Chem. Educ., 53, 418–418.
  38. Pickering M., (1975), Helping High-Risk Freshman Chemist, J. Chem. Educ., 52, 512–514.
  39. Pinarbasi T., Sozbilir M. and Canpolat N., (2009), Prospective chemistry teachers' misconceptions about colligative properties: boiling point elevation and freezing point depression, Chem. Educ. Res. Pract., 10, 273–280.
  40. Reid S. A., (2016), A flipped classroom redesign in general chemistry, Chem. Educ. Res. Pract., 17, 914–922.
  41. Rein K. S. and Brookes D. T., (2015), Student Response to a Partial Inversion of an Organic Chemistry Course for Non-Chemistry Majors, J. Chem. Educ., 92, 797–802.
  42. Repice M. D., Sawyer R. K., Hogrebe M. C., Brown P. L., Luesse S. B., Gealy D. J. and Frey R. F., (2016), Talking through the problems: a study of discourse in peer-led small groups, Chem. Educ. Res. Pract., 17, 555–568.
  43. Robert J., Lewis S. E., Oueini R. and Mapugay A., (2016), Coordinated Implementation and Evaluation of Flipped Classes and Peer-Led Team Learning in General Chemistry, J. Chem. Educ., 93, 1993–1998.
  44. Rootman-le Grange I. and Blackie M. A. L., (2018), Assessing assessment: in pursuit of meaningful learning, Chem. Educ. Res. Pract., 19, 484–490.
  45. Ryan M. D. and Reid S. A., (2016), Impact of the Flipped Classroom on Student Performance and Retention: A Parallel Controlled Study in General Chemistry, J. Chem. Educ., 93, 13–23.
  46. Sanger M. J. and Greenbowe T. J., (1997), Common student misconceptions in electrochemistry: Galvanic, electrolytic, and concentration cells, J. Res. Sci. Teach., 34, 377–398.
  47. Scerri E., (2003), Constructivism, relativism, and chemical education, Ann. N. Y. Acad. Sci., 988, 359–369.
  48. Shields S. P., Hogrebe M. C., Spees W. M., Handlin L. B., Noelken G. P., Riley J. M. and Frey R. F., (2012), A Transition Program for Underprepared Students in General Chemistry: Diagnosis, Implementation, and Evaluation, J. Chem. Educ., 89, 995–1000.
  49. Smith J., Wilson S. B., Banks J., Zhu L. and Varma-Nelson P., (2014), Replicating Peer-Led Team Learning in Cyberspace: Research, Opportunities, and Challenges, J. Res. Sci. Teach., 51, 714–740.
  50. Spencer H. E., (1996), Mathematical SAT test scores and college chemistry grades, J. Chem. Educ., 73, 1150–1153.
  51. Stanich C. A., Pelch M. A., Theobald E. J. and Freeman S., (2018), A new approach to supplementary instruction narrows achievement and affect gaps for underrepresented minorities, first-generation students, and women, Chem. Educ. Res. Pract., 19, 846–866.
  52. Street C. D., Koff R., Fields H., Kuehne L., Handlin L., Getty M. and Parker D. R., (2012), Expanding Access to STEM for At-Risk Learners: A New Application of Universal Design for Instruction, J. Postsecond. Educ. Dis., 25, 363–375.
  53. Taber K. S., (2010), Straw Men and False Dichotomies: Overcoming Philosophical Confusion in Chemical Education, J. Chem. Educ., 87, 552–558.
  54. Villafane S. M., Loertscher J., Minderhout V. and Lewis J. E., (2011), Uncovering students' incorrect ideas about foundational concepts for biochemistry, Chem. Educ. Res. Pract., 12, 210–218.
  55. Vyas V. S., Kemp B. and Reid S. A., (2020), Zeroing In on the Best Early-Course Metrics to Identify At-Risk Students in General Chemistry: An Adaptive Learning Pre-assessment vs. Traditional Diagnostic Exam, unpublished.
  56. Williams J. L., Miller M. E., Avitabile B. C., Burrow D. L., Schmittou A. N., Mann M. K. and Hiatt L. A., (2017), Teaching Students To Be Instrumental in Analysis: Peer-Led Team Learning in the Instrumental Laboratory, J. Chem. Educ., 94, 1889–1895.
  57. Willson-Conrad A. and Kowalske M. G., (2018), Using self-efficacy beliefs to understand how students in a general chemistry course approach the exam process, Chem. Educ. Res. Pract., 19, 265–275.
  58. Wilson S. B. and Varma-Nelson P., (2016), Small Groups, Significant Impact: A Review of Peer-Led Team Learning Research with Implications for STEM Education Researchers and Faculty, J. Chem. Educ., 93, 1686–1702.
  59. Wink D. J., (2014), Constructivist Frameworks in Chemistry Education and the Problem of the “Thumb in the Eye”, J. Chem. Educ., 91, 617–622.
  60. Xu X. Y. and Lewis J. E., (2011), Refinement of a Chemistry Attitude Measure for College Students, J. Chem. Educ., 88, 561–568.
  61. Zoller U., (1990), Students Misunderstandings and Misconceptions in College Freshman Chemistry (General and Organic) - Comment, J. Res. Sci. Teach., 27, 1053–1065.
  62. Zusho A., Pintrich P. R. and Coppola B., (2003), Skill and will: the role of motivation and cognition in the learning of college chemistry, Int. J. Sci. Educ., 25, 1081–1094.

Footnote

† Electronic supplementary information (ESI) available: Four tables and one figure of additional content. Table S1 provides a course key, while Table S2 shows the list of topics included in the prep module. Table S3 provides descriptive statistics for the ALEKS learning rate (topics per hour) for two student cohorts based upon PM completion. Table S4 present correlations of the course grades across general and organic chemistry with PM completion and PQPA. Fig. S1 displays a plot of PM completion vs. time spent in the ALEKS PM. Finally, the quick-start/FAQ guide and survey instrument used in this work. See DOI: 10.1039/d0rp00283f
This journal is © The Royal Society of Chemistry 2021