Yancey D.
Mitchell
,
Jessica
Ippolito
and
Scott E.
Lewis
*
Department of Chemistry & Biochemistry, Kennesaw State University, 1000 Chastain Rd., MB 1203 Kennesaw, GA 30144, USA. E-mail: slewis57@kennesaw.edu; Tel: 678-797-2377
First published on 26th June 2012
Peer-Led Team Learning (PLTL) is a widely disseminated pedagogical reform that employs previously successful undergraduate students, peer leaders, to lead sessions of structured group work in the target class. Numerous studies have evaluated the impact of this reform in various post-secondary chemistry classes. Results from these studies suggest that PLTL may be effective at improving student success in these classes, either through improved performance on common exams or reduced student attrition in the classes. This study seeks to take a broader picture at measuring the impact, by examining the role PLTL plays across a two semester General Chemistry sequence. This includes an analysis of PLTL on students' decision to progress through the two semester sequence, and on PLTL impact on the algorithm-heavy second-semester General Chemistry. The findings suggest that the PLTL implementation is robust in improving student success directly in terms of the target class. However, PLTL had little to no effect on students' decision to continue in the General Chemistry sequence. Additionally, PLTL had little effect on student performance in subsequent courses where the pedagogy returned to lecture-only instruction. The results suggest that PLTL implementation on one course within a sequence would have limited impact, and in order to improve student progress toward graduation, PLTL implementation may have to be curricular wide.
Alternatively, PLTL, or reforms that are largely similar, have been offered as an optional add-on to the course. In this implementation, participation in PLTL is voluntary, and for the participants represents additional structured time-on-task in the setting. In one multi-year study, participants in the PLTL session had a 69% pass rate compared to a 54% pass rate for non-participants (Baez-Galib et al., 2005). A different study with a similar implementation found that participants out-performed non-participants by approximately one-third of a grade point (Hockings et al., 2008). Both of these studies have some indications that the participants were comparable to the non-participants based on incoming cognitive matters, but it remains possible that the participating students had substantially more motivation to succeed than the non-participants.
PLTL has also been evaluated in other chemistry courses. In an Allied Health Chemistry course, mandatory 2 h weekly PLTL sessions were used in comparison to 1 h weekly traditional recitation sessions; student pass rates with the PLTL method improved from 59% to 83% (Akinyele, 2010). With Organic Chemistry, one study replaced recitation sessions with mandatory peer-led sessions. The students in the PLTL session out-performed a historical comparison of overall exam scores by approximately 0.3 standard deviations (Tien et al., 2002). PLTL has also been run in Organic Chemistry as an option for participants. Wamser found that students attending optional PLTL sessions for Organic Chemistry had an 85% pass rate compared to the 69% for those who did not attend (2006). Additionally, students attending PLTL sessions out-performed the non-participants on a standardized end-of-course exam, and featured greater persistence through their three-semester organic sequence. The concern of student motivation in evaluating optional PLTL sessions is also applicable here.
The second aspect of progression is to investigate the role of PLTL on student performance in a subsequent chemistry course. This can provide insight into the ability of PLTL to promote transfer, the application of knowledge and skills to a new situation. For example, it is possible that PLTL improves students' skills in assimilating new chemistry knowledge, and these skills could then be transferred to subsequent chemistry courses where continued improved success would be witnessed. Alternatively, PLTL may only facilitate students' assimilation of knowledge while the pedagogy is enacted, and student success returns to normal in subsequent courses. Ultimately, addressing the ability of PLTL to promote transfer can suggest whether implementing the reform in one targeted course is sufficient to enable continued student success or whether PLTL should be recommended for curriculum-wide adoption in order to promote student success.
With the above topic splitting, second-semester General Chemistry has a much heavier reliance on complex algorithms. For example, in determining the pH of a basic salt, given the molarity of the salt and the acid-dissociation constant of the conjugate weak-acid, students are required to convert the acid-dissociation constant into the base-dissociation constant, develop an equation to describe the equilibrium conditions, solve for the molarity of the hydroxide ion, and then convert to pH. Other topics, such as equilibrium calculations and the interchange of equilibrium, free energy and electrochemistry, can offer similarly involved algorithms. As a result, the implementation of PLTL in second-semester General Chemistry faces unique challenges compared to the implementation in first-semester General Chemistry. Identifying and training suitable peer-leaders with the necessary math skills and content knowledge is the most apparent new challenge, but also facilitating successful student interaction on complex material in a limited time-frame poses difficulties. Given these challenges, which can be unique to second-semester General Chemistry, evaluating PLTL's success with this course is a necessary step toward recommending curriculum-wide adoption of the pedagogy.
• What is the effect of implementing PLTL in first-semester General Chemistry on students' decision to enroll in the subsequent follow-on chemistry course, second-semester General Chemistry?
• What is the effect of implementing PLTL in first-semester General Chemistry on students' performance in the follow-on course?
• What is the effect of implementing PLTL in second-semester General Chemistry on students' performance in this course?
The classes did share a common final exam created by the American Chemical Society (ACS). GC1 used the First-Term General Chemistry Exam (2002) until Fall 2010 when there was a switch to the First-Term General Chemistry Exam (2009). As a result, students' ACS GC1 exam scores were standardized for each version of the test, thus a score of 1.00 would represent one standard deviation above the average exam score within the setting. GC2 used the Second-Term General Chemistry Exam (2006) for the duration of the study and scores represent the number of questions correct with a range of 0 to 70 possible. Additionally, students' demographics, college entrance scores (SAT scores) and final letter grade were obtained from university records. All of the data collected in the study was with the approval of the university's Institutional Review Board.
Data was collected for GC2 students from the Fall 2009 semester through the Spring 2011 semester. During this timeframe, 24 classes of GC2 were offered, with 8 of those utilizing the PLTL reform. Data was also collected on GC1 students from the Spring 2009 semester through the Fall 2010 semester, which precedes the GC2 timeframe by one semester, to model the progression of students from GC1 to GC2. Within the GC1 timeframe 35 classes of GC1 were offered and 12 of those used the PLTL reform. All of the statistical analyses performed were conducted at the class-level to maintain independence of observation and were conducted with an alpha-level of 0.05. The effect sizes for any significant differences are described with Cohen's d, where a value of 0.8 and above represents a large effect size (Cohen, 1988).
Traditionally in the research setting, GC1 and GC2 are taught relying primarily on lectures, meeting twice weekly in 75 min lectures. With the PLTL reform, the two weekly lectures are shortened to 50 min each, and once a week a 50 min peer-led problem solving session is incorporated, thus maintaining an equal amount of class time. Attendance to the problem-solving session is a mandatory part of the course for students, and constituted 12% of their overall grade in the course (as a portion of the 20% that was the instructors' discretion). During the problem solving session, students work in groups of four on chemistry problems. The problems are designed by the course instructors to relate to material recently covered in the course. The problem solving sessions are conducted by a peer-leader. Separate classrooms were secured for the problem solving sessions, and either one peer leader worked with 12 to 16 students or two peer leaders worked together with 20 to 24 students.
The role of the peer-leader is to promote cooperative learning among the students, to encourage all students to participate in the problem-solving session, to direct students to resources when they are stuck and to challenge students to demonstrate their understanding of the problems both verbally and in writing. To prepare peer-leaders for their role, they meet in a weekly training course that is run as a mock peer-led session. The peer-leaders work in groups on the same problems their students will receive that week, and the instructor models the role of the peer-leader. Peer-leaders receive two credit-hours of upper-level elective credit for completing the training course. As part of the training, peer leaders were required to keep a reflective journal and perform periodic observations of other peer leaders. In these observations, peer leaders complete an observation rubric. In this rubric, the observing peer-leader notes the types of interactions among students and also describes how the observed peer-leader addresses the students. The intent is that this will provide feedback on the extent group work is facilitated in their session. Additionally, the observing peer-leader makes constructive suggestions for improvement. Peer-leaders are also observed throughout the semester by a course instructor to provide constructive feedback for promoting the cooperative learning environment. More details on the peer-led problem-solving session, peer-leader training and the role of the instructors have been described previously (Lewis, 2011).
The improved Pass GC1 rate with PLTL (t = 4.2, d = 1.6) was expected from previous research. The Enroll GC2 rate, though, was not appreciably improved through PLTL. Thus the implementation of PLTL in GC1, with its expressed reliance on cooperative learning, did not appear to impact students' decision to enroll in the subsequent course. It is worth noting that the Pass GC1 rate and Enroll GC2 rates when combined did indicate that PLTL with GC1 placed more students into GC2, though only through the improved pass rate.
The overall Enroll GC2 rate in the setting was 64% at the class-level, or 63.1% at the student-level. This indicates that 36.9% of those students passing GC1, do not take GC2, in the timeframe for the study. As all majors in the setting that require completion of GC1 also require the completion of GC2, it is reasonable to conclude that these students either transferred from the setting or elected to change their major. To investigate the proportion of student transfers, students' last term enrolled at the setting was compared with the term in which they took GC1. It was found that 6.2% of those who passed GC1 in the setting were not enrolled in future terms. Thus 30.7% of those who pass GC1 did enroll in future terms but did not take GC2. Ultimately, this rate indicates a sizable hindrance to student progression in the science-disciplines that is similar in scale to the GC1 success rate.
Traditional GC1 (Std. Dev.) | PLTL in GC1 (Std. Dev.) | |
---|---|---|
SAT Math Average | 559 (69) | 555 (65) |
SAT Verbal Average | 544 (69) | 537 (64) |
GC1 ACS Standardized | 0.32 (0.90) | 0.26 (0.89) |
Second, it may be that the PLTL pedagogy in GC1 did not adequately prepare students to enter GC2. However, the evidence collected so far has shown that the PLTL pedagogy with GC1 led to comparable GC1 ACS scores with the traditional pedagogy which may serve as evidence that PLTL did adequately prepare students for entering GC2 (Lewis, 2011). It is worth noting that the student-level correlation between standardized GC1 ACS and GC2 ACS was 0.581 at the setting, indicating the standardized GC1 ACS is a reasonable measure for describing GC2 preparation.
A third possibility for the difference in pass rates may be that the change in pedagogy to traditional GC2 instruction had adverse effects on student success in the GC2 course. There is some evidence to support this possibility. Among those students who were in PLTL in GC1, 53.3% of them enrolled in PLTL in GC2. For those students who enrolled in PLTL in GC2 there was observed a 79.3% GC2 Pass Rate. For those students who enrolled in PLTL in GC1 but then enrolled in traditional GC2 instruction, the GC2 Pass Rate was 63.0%. Among the four possibilities, students who changed from PLTL GC1 to traditional GC2 had the lowest pass rate observed, as shown in Table 3. It appears, therefore, that the improved success observed with PLTL in GC1 did not lead to transferable skills that students could then employ in GC2, and rather the effectiveness of PLTL is concentrated on the course in which it is enacted.
GC1 Instruction | GC2 Instruction | N | GC2 Pass Rate |
---|---|---|---|
PLTL | PLTL | 188 | 79.3% |
PLTL | Traditional | 165 | 63.0% |
Traditional | PLTL | 144 | 81.3% |
Traditional | Traditional | 344 | 76.7% |
The pass rates described in Table 3 may lead one to the conclusion that the most appropriate course of action is to offer traditional instruction in GC1 and PLTL in GC2. However, as Table 3 indicates only students enrolled in GC2, such a conclusion does not account for the improved GC1 Pass Rate with PLTL in GC1 (see Table 1). With this consideration, the most appropriate comparisons within Table 3 are between those rows with the same GC1 Instruction.
To ultimately discern the impact of PLTL in GC1 on student progression through the General Chemistry sequence, the progression rates (Table 1) can be compared. It is found that a GC1 class using PLTL has a higher percent of students successfully completing GC2 (35% versus 30%). This benefit can be entirely attributed to the improved pass rate in GC1, and is mitigated by the pass rate in GC2. The data in Table 3 suggests that the incorporation of PLTL in GC2 would improve the pass rate in GC2, and thereby offer a stronger benefit for student progression, than only implementing PLTL in GC1. The impact of PLTL in GC2 on GC2 pass rates is more formally addressed next.
Variable | Traditional (Std. Dev.) | PLTL (Std. Dev.) |
---|---|---|
a Statistically significant difference, p < 0.05. | ||
Number of classes | 12 | 8 |
Average class size | 60 (12) | 54 (15) |
Average on GC2 ACS Exam | 34 (2) | 34 (2) |
Percent Taking GC2 ACS Exama | 79% (7%) | 91% (6%) |
Percent Passing the Classa | 57% (7%) | 70% (11%) |
SAT Math Average | 559 (12) | 558 (11) |
SAT Verbal Average | 535 (11) | 541 (12) |
GC1 ACS Standardized | 0.27 (0.14) | 0.22 (0.20) |
From the results in Table 4, there was a statistically significant difference between the pedagogies on the percent that took the GC2 ACS Exam (t = 3.87, d = 1.78). Similarly, there was a significant difference in the percent that passed the class (t = 3.24, d = 1.41). These measures each represent student retention: the percent taking the GC2 ACS Exam represents an inverse measure of student withdrawals, and the percent passing the class represents an inverse measure of students withdrawing from or failing the course. Ultimately, GC2 classes employing PLTL featured noticeably lower student withdrawal rates.
The improvement in student retention with the PLTL pedagogy is impressive but should also be considered in the context of the other variables measured. No statistically significant difference was found between the two sets of classes in terms of pre-semester measures of student performance: SAT Math, SAT Verbal and the GC1 ACS score. However, the failure to find statistical significance does not indicate the two groups are equivalent on these measures. Particularly with the small sample size, there may be insufficient power to statistically indicate differences (Lewis and Lewis, 2005b). Using the two-one sided t-test suggested in this reference, equivalence between these groups could not be supported on these measures using an interval of ±0.5 of the pooled standard deviation, again likely owing to the sample size. To rule out the impact of these pre-semester variables, a regression model to predict GC2 ACS Take with PLTL, SAT Math, SAT Verbal and GC1 ACS as predictors was run. PLTL was the only predictor variable found to remain statistically significant, with a coefficient of 11.2 close to the observed difference on this outcome. Similarly, a regression model to predict the percent passing found only PLTL to be significant, with a coefficient of 13.9, also close to the observed difference in pass rate. In summary, while the sample size is insufficient to claim the groups are equivalent on the pre-semester measures, there is no evidence that suggests the pre-semester measures are linked to the differences observed on GC2 ACS Take or the pass rate.
It is also noteworthy that there is no statistically significant difference on the average GC2 ACS Exam score. As this GC2 ACS Exam is a cumulative exam designed to measure students' knowledge of the course material, the comparable performance of the pedagogies suggests that the improvement in student retention with PLTL does not come at the expense of lowering standards within the course. Interestingly, these results mirror the impact the PLTL pedagogy had on GC1 classes, which also saw an improvement in student retention in the 10 to 15% range while maintaining performance on the common final (Lewis, 2011).
An additional concern for the comparison of the two pedagogies is the possible instructor effect on the results. That is, if instructors who elect to teach with the PLTL model are in some meaningful way different than the instructors using the traditional approach, this could be responsible for the difference observed. To address this concern, a subset of the instructors had taught both pedagogies within the time frame of the study. A comparison of only the classes from the common instructors is shown in Table 5.
Traditional (Std. Dev.) | PLTL (Std. Dev.) | |
---|---|---|
Number of classes | 4 | 4 |
Average class size | 65 (11) | 66 (10) |
Average on ACS Exam | 34 (1) | 34 (1) |
Percent Taking ACS Exam | 81% (5%) | 90% (7%) |
Percent Passing the Class | 60% (11%) | 72% (14%) |
SAT Math Average | 564 (5) | 551 (7) |
SAT Verbal Average | 537 (12) | 551 (7) |
GC1 ACS Standardized | 0.24 (0.10) | 0.10 (0.21) |
As the common instructor comparison featured only four classes from each pedagogy, there is insufficient statistical power to demonstrate a statistically significant difference on the outcome measures. Qualitatively, the improvement in the student retention measures with PLTL is similar to the results of the whole sample, suggesting that the observed differences can be attributed to the pedagogy and not a result of instructor effect.
Students from the PLTL reform in GC1 were found to have a lower pass rate in GC2 than students in traditional GC1 instruction, though the difference was not found to be statistically significant. The reason for this decline that is best supported by the data is the change in pedagogy, as students who switched from PLTL GC1 into traditional GC2 fared worse than those who stayed with the PLTL pedagogy in GC2. This finding suggests that one semester of the PLTL pedagogy in the setting did not offer skills that are transferrable to a subsequent course; instead it appears that PLTL facilitates students' knowledge acquisition primarily while they are participants in the pedagogy. This finding also bears relevance for the issue of student attrition, as the implementation of PLTL in one target course would likely have limited impact on students' completing science degrees. Instead, the implementation of PLTL across the curriculum may be necessary to substantially improve students' graduation rates in the sciences, though this hypothesis remains untested.
Evidence from this study indicates that PLTL implementation into GC2 does feature improved student success similar to previous studies with GC1 coursework (Baez-Galib et al., 2005; Lewis, 2011). Classes employing PLTL in GC2 featured a 70.2% pass rate compared to a 57.1% pass rate with traditional classes, while maintaining comparable performance on the ACS final exam. It is argued that this is evidence of PLTL facilitation of students' knowledge acquisition, as more students were able to achieve passing scores with the reform pedagogy than with traditional instruction. Ultimately, the results of this study support the adoption (and further evaluation) of PLTL as a curriculum-wide intervention. As a reform targeting a specific course, it is found that PLTL's success is mostly limited to the target course, and that the return to traditional pedagogy can be detrimental. However, this study and the previously cited studies have now documented improved student performance when PLTL is implemented in either General Chemistry or Organic Chemistry. A curriculum-wide adoption of PLTL, then, may substantially improve students' graduation rates in chemistry.
This journal is © The Royal Society of Chemistry 2012 |