Scott A.
Reid
*a,
Laura
MacBride
b,
Llanie
Nobile
a,
Adam T.
Fiedler
a and
James R.
Gardinier
a
aDepartment of Chemistry, Marquette University, Milwaukee, WI 53233, USA. E-mail: scott.reid@marquette.edu
bOffice of Institutional Research, Marquette University, Milwaukee, WI 53233, USA
First published on 22nd December 2020
General chemistry courses are key gateways for many Science, Technology, Engineering, and Mathematics (STEM) majors. Here, we report on the implementation and evaluation of an adaptive, ALEKS-based online preparatory module (PM) for general chemistry. The module was made available in Summer 2018 at no cost to all students entering any section of general chemistry that fall. Of the 827 students who registered into the PM, 44% fully completed the module, 48% completed part of the module, and 8% did not complete any of the module. Considering students enrolled in first-term general chemistry, we find a marked increase in ACS final exam percentile for students who completed more than 50% of the module. This is suggested to reflect the self-selection of users who were highly motivated and/or likely to succeed in the course, a hypothesis supported by an analysis using an internal diagnostic metric, the predicted first-year quality point average (PQPA). To examine longer term impacts of the PM, we examined performance in subsequent chemistry courses, through second-semester organic, and found that students completing more than 50% of the module outperformed their counterparts across all courses, with the gap largest in first semester general chemistry and narrowing across subsequent courses. Finally, we surveyed students in summer 2020, two years after the PM offering. The survey indicated overall satisfaction with the PM. For students who did not complete the module, primary reasons given were difficulty (29% of respondents) and insufficient time (46%). As the module did not proportionally benefit the target group of underprepared or at-risk students, we suggest tweaks for future implementations.
A follow-up to the Texas Tech study involved a survey of students who completed the remedial course, but did not continue farther (Jones and Gellene, 2005). The primary reason identified was a change of major or degree program, with the importance of the remedial course experience being grade dependent. For students receiving an A in the remedial course, ∼10% indicated that the course was a factor in their decision, while for students receiving a C, this percentage increased to ∼50%. The latter group also gave a neutral overall rating to the course.
Studies like these have raised the important point of whether preparatory or remedial courses provide any benefit for at-risk student populations. At worst, such courses may act to discourage students from pursuing STEM degree paths, to the degree that they impact affective domain characteristics such as self-efficacy, motivation, attitude, and interest, which are known contributors to student success (House, 1995; Chemers, et al., 2001; Zusho, et al., 2003; Bauer, 2005; Dalgety and Coll, 2006; Ayan, 2010; Galyon, et al., 2011; Lynch and Trujillo, 2011; Ferrell and Barbera, 2015; Nielsen and Yezierski, 2015; Chan and Bauer, 2016; Willson-Conrad and Kowalske, 2018). For example, recent studies in GC courses have shown a connection between student performance on exams and their self-efficacy beliefs (Willson-Conrad and Kowalske, 2018). The importance of attitude in identifying at-risk students and explaining variation in chemistry achievement has also been demonstrated (Bauer, 2005; Bauer, 2008; Brandriet, et al., 2011; Xu and Lewis, 2011; Chan and Bauer, 2014).
Additional insight into the efficacy of preparatory courses comes from the study of a voluntary, self-paced online summer preparatory course for GC (Botch, et al., 2007). A comparison of non-users vs. users who completed more than 50% of the course showed striking differences in final course grades in the first semester course. The authors identified three different student types: (1) those underprepared and unlikely to succeed without additional help, (2) those likely to succeed but considering themselves underprepared, and (3) those likely to succeed and who make use of all available resources. Although the preparatory course was designed for group 1, the authors concluded that the majority of users came from groups 2 and 3. For these students, the preparatory course proved a useful tool, yet the target audience of weaker or less prepared students did not take advantage of the course in proportionate numbers. Consistent with this finding, a recent study of a voluntary online preparatory course for organic chemistry showed that the academic preparation of students was a significant factor in predicting their participation in the course (Fischer, et al., 2019). A similar finding was reported in the study of peer-led team learning (PLTL) groups, where again voluntary participation was suggested to lead to differential participation by more highly motivated students (Chan and Bauer, 2015).
Studies to date have typically examined the utility of prep courses or modules in governing success in the introductory (first-term) course; however, it is also desirable to examine performance in subsequent chemistry courses, through organic chemistry. As part of a larger effort to develop a successful intervention program at our institution, we report here a study of the effect of a voluntary adaptive summer prep module (PM) based on ALEKS. ALEKS, or Adaptive LEarning in Knowledge Spaces, is an adaptive program developed by the ALEKS Corporation (https://www.aleks.com) which is widely used in introductory chemistry, math, and physics. The module was made available by the vendor at no-cost to all students enrolled in general chemistry courses in Fall 2018, and opened shortly following the close of the summer registration period, roughly 6 weeks prior to the start of the fall term. We first examine correlations between final (ACS) exam performance and PM completion for all students who entered the first-term non-majors general chemistry course (N = 676). We then probe correlations with predicted first semester quality point average (PQPA), an internal metric described in detail below, for all first-year students in the cohort for which PQPA data was available (N = 510). We subsequently analyze, for the full cohort, performance and completion rates in subsequent courses including second-term general chemistry and organic chemistry. Finally, we report the results of a student survey administered two years following the offering of the PM.
Thus, the research questions which we sought to address here are as follows: (1) What is the effect of an online adaptive, voluntary summer PM on student performance in first-semester general chemistry? (2) What is the longitudinal effect, including attrition rate, of the prep course on performance across second-semester general chemistry and organic chemistry? (3) What is the efficacy of the PM for at-risk students, as identified by institutional metrics? and (4) For students who did not complete the PM, what factors were most important?
For the summer prep implementation used here, students were only required to learn the complete set of topics to complete the course, no periodic re-checks of knowledge were scheduled. The module consisted of 137 topics spanning basic mathematics and physics to specific chemistry knowledge (Table S2 shows included topics, ESI†). At the close of the final summer registration session for new first-year students, in early July 2018, an email invitation was sent, containing a detailed Quick-start/FAQ sheet (ESI†), to all students enrolled in first-term or second-term (off-semester) general chemistry in Fall semester 2018. The preparatory course was opened for enrollment in mid-July, and kept open until the end of the second week of fall classes (early September). The composition of the GC1-nonM (Table S1, ESI†) sections by major was ∼25% engineering (first- or second-year depending on major), ∼45% health sciences, ∼15% biology, with all other majors comprising ∼15%.
We examined bivariate correlations of preparatory module completion with ACS final exam score, and partial correlations controlling for time spent in the module. These correlations were examined for all students who completed the course and attempted the PM, Table 1. The ACS final exam percentile displayed a moderate correlation with PM completion, which increased when controlling for time spent in the PM. If only first semester students (i.e., those with a PQPA in the data set) were considered, the correlations were similar. A similar correlation was also found between the ACS final and PQPA, Table 1, while a weaker correlation was observed with initial ALEKS knowledge check.
Variable 1 | Variable 2 | N | Control | Pearson correlation | Significance level |
---|---|---|---|---|---|
ACS final exam percentile | Preparatory module completion | 676 | None | 0.51 | <0.01 |
Time spent in preparatory module | 0.55 | <0.01 | |||
510 | Students w/PQPA (1st semester) | 0.51 | <0.01 | ||
Time spent in preparatory module | 0.57 | <0.01 | |||
ACS final exam percentile | Initial ALEKS knowledge check | 676 | None | 0.35 | <0.01 |
ACS final exam percentile | PQPA | 510 | None | 0.58 | <0.01 |
Preparatory module completion | PQPA | 578 | None | 0.46 | <0.01 |
Initial ALEKS knowledge check | PQPA | 578 | None | 0.32 | <0.01 |
Following the approach employed by Botch and co-workers, (Botch, et al., 2007) we initially binned students into subgroups by module completion as: (1) 50–100% completion, (2) attempted with partial completion less than 49%, (3) did not attempt. However, the differences in mean final exam score of the latter two groups was not statistically significant, and therefore we report here an analysis using only two groups: (1) 50–100% completion, and (2) 0–49% completion. Table 2 shows the relevant means and the results of an independent samples T-test for these groups, compared both for the full cohort, and first semester students for which PQPA data was available. The difference in means is significant at the p < 0.001 level with positive and large effect sizes.
The data in Table 2 reveals a marked difference in ACS final exam percentile for the two subgroups. We suspect that this trend is due to self-selection of users who were likely to succeed in the course, but either lacked confidence in their preparation or were motivated to make use of every available resource (Botch, et al., 2007). This supposition is reinforced by examining PM completion vs final course grade. For students (N = 66) who received a final first-semester general chemistry course grade of D, F or W (withdrawal), only 45% completed more than 50% of the PM, and only 12% fully completed the module. In comparison, of students who received a final course grade of C– or higher (N = 662), 80% completed more than 50% of the PM, and 48% fully completed the module.
Probing further, we performed an independent samples T-test with PQPA as the controlling variable. Setting a cutoff PQPA of 2.80, a cut point recently validated for identifying at-risk students, (Vyas, et al., 2020) a comparison of PM completion for the two subgroups is shown in Table 3. The mean difference in completion was ∼36% (p < 0.001), with a large effect size. If we instead consider students who partially or completely finished the PM (N = 465) vs. those who enrolled into the PM but did not take the initial knowledge check (N = 45), we find a mean difference in PQPA of 0.25 (p < 0.001, effect size of 0.86). This reinforces our supposition that students who were at-risk or least prepared did not take advantage of the PM to the same degree as other students.
In a subsequent phase of analysis, we examined the longitudinal performance of first year students in the PM through the first two-years of college chemistry (general and organic). For analysis of course grades, we kept the same binning for PM completion, and binned students by PQPA into three distinct sub-bins (PQPA = 2.00–3.00, 3.01–3.50, >3.50) for comparison. For students in each bin, we calculated the mean QPA across four courses; first and second term non-majors general chemistry (GC1-nonM/GC2-nonM; Table S1, ESI†), and first and second term organic chemistry (OC1-nonM/OC2-nonM; Table S1, ESI†). For students retaking the courses, we only analyzed the result of their initial attempt. Fig. 1 shows the complete results, where red is consistently used to label the group completing less than 50% of the prep module, and blue the group completing 50% or more. In each panel, bars indicate the number of students, while markers indicate the mean QPA, shown with one standard deviation error bars. Black arrows mark the y-axis associated with each data set.
Fig. 1 shows that across all PQPA groups and all courses, students completing at least 50% of the PM received higher grades; however, the difference between these groups typically narrowed across the course sequence. This trend is less meaningful for the highest PQPA group due to the very small number of students who did not complete the PM. However, in the lowest PQPA group, the difference in QPA between subgroups was roughly 1 unit for GC1-nonM, and narrowed to ∼0.2 unit for the final course in the sequence, OC2-nonM. Consistent with our finding above that less well-prepared students did not take advantage of the prep module to the same degree as other students, the prep module groupings are nearly equal for the lowest PQPA cohort, while for the middle and highest PQPA cohorts the group completing at least 50% of the prep module was dominant.
Recently, Freeman and co-workers examined course data over a 15-year period for students in general and organic chemistry at the University of Washington (Harris, et al., 2020). It was reported that underrepresented students performed more poorly in the first semester general chemistry course, and those obtaining low grades were more likely to drop out of STEM majors. However, those obtaining sufficient grades to move on in the sequence were more likely to continue than their peers. Similarly, we find (Fig. 1) that the differential in course grades between students in the lowest and highest PQPA cohort significantly decreased over the sequence of courses. For example, in GC1-nonM the mean course QPA by PQPA cohort was: 2.14 (lowest); 2.99 (middle); 3.64 (highest). By the final course in the sequence, OC2-nonM, the gap in mean course QPA had significantly narrowed (lowest = 3.11; middle = 3.31; highest = 3.65).
These trends could be influenced by demographics, and by selective attrition. Considering demographics, the lowest PQPA cohort had its highest population of students from the College of Arts and Sciences (68%), while the middle and highest PQPA cohorts were dominated by students from Health Sciences (66% and 74%, respectively), which contain a large population of pre-professional students. However, Arts and Science majors included those which require two years of chemistry (e.g., Biology, Biochemistry, Chemistry), as well as those not requiring chemistry at all (e.g., Psychology), where chemistry is taken as preparation for professional school. Thus, all PQPA cohorts included significant populations of students required to take two years of college chemistry for their degree or professional program of interest.
Considering selective attrition, we examined more carefully the attrition rates between first-term and second-term general chemistry and first- and second-term organic chemistry in each subgroup, and this data is presented in Table 4. While the numbers in some subgroups are small, in each subgroup we generally see a smaller attrition rate for students in the higher PM completion group, which may contribute to the trends observed in Fig. 1. This is particularly pronounced in the lowest PQPA group. The exception to these trends, interestingly, is in the highest PQPA group in general chemistry, where a larger attrition rate is observed in the higher PM completion group. Given the very small number of students who completed less that 50% of the PM, this difference may not be significant. Moreover, as the mean course QPA of students in this group was 3.69, any attrition clearly is due to other factors (e.g., a change of major or degree program, delaying the second-term course until summer).
PQPA group | Course sequence | PM completion (%) | N | Attrition rate (%) |
---|---|---|---|---|
High | General (non-majors) | ≥50 | 91 | 30 |
<50 | 6 | 17 | ||
Organic (non-majors) | ≥50 | 49 | 16 | |
<50 | 3 | 33 | ||
Middle | General (non-majors) | ≥50 | 246 | 23 |
<50 | 50 | 32 | ||
Organic (non-majors) | ≥50 | 105 | 25 | |
<50 | 21 | 33 | ||
Low | General (non-majors) | ≥50 | 56 | 36 |
<50 | 59 | 44 | ||
Organic (non-majors) | ≥50 | 17 | 22 | |
<50 | 18 | 42 |
To further examine the longitudinal impact of the PM, we examined Pearson correlations between course grades across general and organic chemistry, PM completion, and PQPA. These are presented in Table S4 (ESI†). Perhaps unsurprisingly, the correlations with PM completion and PQPA, which are all significant at the p = 0.01 level, are highest in first-term general chemistry and drop steadily thereafter. However, the drop in correlation with PM completion from first-term general chemistry to second-term organic (∼15%) in noticeably smaller than that with PQPA (37%).
The data presented thus far show that: (1) completion of a summer online prep module is correlated with success in first-term general chemistry and beyond, (2) students taking advantage of the course were primarily those who were already prepared and highly motivated to succeed, with underprepared or at-risk students not taking advantage of the course to the same degree, (3) the differential in course grades between underprepared and well-prepared students significantly decreased over the sequence of courses from first-term general chemistry to second-term organic, while (4) the attrition rate overall was larger for less well-prepared students as measured by PM completion. To better understand these trends, we turn to the results of the student survey.
The survey was administered electronically. An invitation to the survey (instrument provided in the ESI†) was sent via email to 827 students, with a follow-up reminder sent roughly two weeks later. Ultimately, 255 responses were received (31% response rate). Unsurprisingly, the demographics of survey respondents were significantly different from that of the entire set, with 76% of survey respondents female (59% for the entire set) and 64% of respondents indicating that they had fully completed the preparatory module (44% for the entire set). In Fig. 2, we show the mean results from a response to seven questions concerning the preparatory module, with responses on a 5-point Likert scale from 1 (strongly agree) to 5 (strongly disagree). It is noteworthy that respondents were neutral on the questions of whether, looking back, they should have spent more time in the module (Q4) or whether the module was a benefit in subsequent courses (Q7). The logic for including the latter question was that, while the majority of PM topics covered first-semester material (see Table S2 in the ESI†), many of these topics (e.g., dimensional analysis, chemical nomenclature, fundamental reactivity and energy concepts) can be considered foundational to success in chemistry. Thus, the goal of this question was to capture student perceptions of the longitudinal effect of the module, given trends in the data which show disparities between PM completion subgroups that persist across multiple courses.
![]() | ||
Fig. 2 Survey questions and responses on a 5 point Likert scale, from 1 (strongly agree) to 5 (strongly disagree). |
The majority of participants agreed that the module was helpful (Q1) and that they would recommend it to other students (Q6). In our offering of the module, help outside of the ALEKS program (e.g., virtual instructor or TA office hours) was not made available; however, a majority of students agreed that such help would be desirable (Q5).
A separate section of the survey focused on students who had not fully completed the module – here, students were asked to identify the most important factor(s) which hindered their completion of the module. Interestingly, 29% of respondents indicated that the preparatory module was too difficult, while 46% indicated that they did not have sufficient time to complete the module due to other activities (work, etc.). Another 12% of respondents indicated that they either chose not to participate or already considered themselves well prepared for college level chemistry.
In light of these responses, we note that the initial ALEKS knowledge check (topic mastery) of students who ultimately completed more than 50% of the PM was 33%, in comparison with 22% for students who ultimately completed less than 50% of the PM. However, we found that the learning rate of students who completed only part of the module was roughly twice that of students who fully completed the module. Thus, as there were no restrictions on the ordering of topic completion, it is plausible that students who found the module difficult overall were successful in completing some topics but were challenged by others. Moreover, as the time required to complete the module was larger for students who began with lower initial mastery, the time available to them may have been insufficient to complete all the PM. In that vein, we note that the mean completion time for students completing 100% of the PM was roughly 27 hours. As the PM was opened in early July and kept open for roughly 8 weeks, this amounted to roughly 3 hours of work per week.
The survey responses also highlight possible unintended consequences of the PM. While it is designed to ensure that all students are well prepared for the course, students who fail to complete the PM may experience additional anxiety or doubt. Such anxiety may also be felt by students who are time challenged to complete the module due, e.g., to full-time summer employment. In our implementation, we extended the due date for the preparatory module two weeks into the fall term, in order to provide additional support for students. This had the positive benefit of significantly increasing attendance at early semester office hours, which are normally not heavily populated. However, students struggling to complete the module may not have taken advantage of this resource.
In closing, we ask: is an adaptive summer prep module for general chemistry of benefit, and if so, whom does it benefit? Our analysis shows that it does benefit students, but not all students, for it is clear that on balance students who need the module the most utilize it less. Thus, the module may only serve to exacerbate the fault lines which already exist between prepared and underprepared students. Moving forward, we consider several tweaks to augment the utility of the module. First, for students who find the module too difficult, or have little time to complete it, a greater emphasis could be directed towards completing some of the module, as opposed to all. This could be achieved by breaking the module into smaller submodules, with students allowed to choose which if any modules to complete, or by awarding credit for completing some fraction, say ½, of the module, with a smaller bonus given for additional completion. A second tweak could provide scaffolded support in the form of virtual instructor/TA office hours in the summer period. Finally, by extending the due date for the module farther into the term, peer led team learning (PLTL) sessions (Mitchell, et al., 2012; Shields, et al., 2012; Street, et al., 2012; Smith, et al., 2014; Rein and Brookes, 2015; Repice, et al., 2016; Robert, et al., 2016; Wilson and Varma-Nelson, 2016; Williams, et al., 2017; Lewis, 2018; Stanich, et al., 2018) could be held in the early part of the semester for students who did not have the opportunity to complete the module in the summer period.
In order to examine the longitudinal effect of the prep module, we examined course grades in subsequent chemistry courses through second-term organic, and conducted a survey of students two years after the offering of the module. Longitudinally, we find that students completing at least 50% of the module outperformed their peers across all chemistry courses in the sequence. However, the gap did narrow across the sequence of courses. Results of the student survey showed that overall, students were satisfied with the prep module. For students who were not able to complete the module, the primary factors identified were the difficulty of the module and a lack of time. Moving forward, implementation of the PM must include strategies to ensure the participation of, and benefit to, the target population of at-risk or underprepared students.
Footnote |
† Electronic supplementary information (ESI) available: Four tables and one figure of additional content. Table S1 provides a course key, while Table S2 shows the list of topics included in the prep module. Table S3 provides descriptive statistics for the ALEKS learning rate (topics per hour) for two student cohorts based upon PM completion. Table S4 present correlations of the course grades across general and organic chemistry with PM completion and PQPA. Fig. S1 displays a plot of PM completion vs. time spent in the ALEKS PM. Finally, the quick-start/FAQ guide and survey instrument used in this work. See DOI: 10.1039/d0rp00283f |
This journal is © The Royal Society of Chemistry 2021 |