R. Ma.
Gregorius
Department of Chemistry and Biochemistry, Canisius College, 2001 Main Street, Buffalo, NY 14208, USA
First published on 10th August 2017
Student performance in a flipped classroom with an animation-based content knowledge development system for the bottom third of the incoming first year college students was compared to that in a traditional lecture-based teaching method. 52% of these students withdrew from the traditionally taught General Chemistry course, compared to 22% in a similar course taught in a flipped classroom teaching method. Of the students who persisted in the course and obtained a grade, there was an increase in A's and B's as well as an increase in D's and F's for students taught in a flipped classroom teaching method when compared to those in the traditional setting. When the course that was initially taught in a flipped classroom method reverted to a traditional teaching method, students in that course generally performed worse than students who were in a traditionally taught course all throughout.
A survey of the literature on the flipped classroom teaching method (Logan, 2015; Seery, 2015; Zainuddin and Halili, 2016; DeLozier and Rhodes, 2017) revealed several variants of the classroom activity: quizzes, teacher-led discussions, worksheets to be completed individually, student-group activities, merging the flipped classroom teaching method with Peer-Led Team Learning (PLTL) (Robert et al., 2016), with Just In Time (JIT) teaching methods (Muzyka, 2015) and implementation of “productive failure” designs (Song and Kapur, 2017). Most of the content delivery, however, involved some variant of a printed text reading assignment (Lenczewski, 2016) and, more commonly, the viewing of a video-recorded lecture or screencast (Seery, 2015). The video-recordings were variants of an instructor being recorded while delivering the content in a lecture format. We found one instance (Fitzgerald and Li, 2015) where the Prezi presentation platform, a more robust form of slide presentation, was used, and Ryan and Reid (2016) used a voice-over PowerPoint video captured in Camtasia.
Our previous work on student-centered, self-guided learning showed that a Flash-based animation (Gregorius, 2010) was better at allowing students to develop content knowledge for and by themselves. We found that students at different levels, grades 3 and 4, high school, and first year college students, could effectively teach themselves specific content using carefully calibrated animations and simulations (Gregorius, 2005a, 2005b; Gregorius et al., 2010a, 2010b). Students who developed content knowledge in this way performed better on assessments than their counterparts who learned from printed texts, traditional lectures, or other forms of direct instruction (Gregorius, 2011). We wondered if these animations and simulations would be an effective content delivery system in a flipped classroom teaching method. We decided to collect data on student performance in a flipped classroom wherein the students developed content knowledge from animations and simulations rather than from direct instruction through video-recorded lectures.
A survey of the literature on the use of the flipped classroom teaching methods in chemistry courses showed generally positive outcomes. There were many reports on the effectiveness of a flipped classroom teaching and learning method: in High School Advanced Placement (AP) Chemistry (Schultz et al., 2014), in General Chemistry (Hibbard et al., 2016), specifically with low achieving students in General Chemistry (Ryan and Reid, 2016), with both the upper third and lower third of the students in General Chemistry (Butzler, 2015), and in Organic Chemistry (Rossi, 2015; Mooring et al., 2016; Shattuck, 2016). There were also some reports in which no statistically significant difference in student performance between those taught in the traditional lecture and textbook system versus the flipped classroom system was observed (Weaver, 2015), sometimes within the same article reporting on the benefits of the flipped classroom teaching method and for a specific student population or chemistry topic (Butzler, 2015; Ryan and Reid, 2016; Shattuck, 2016).
At the author's institution, we had a somewhat unique opportunity in that the lower performing third (to as much as the lower half) of all students taking the first semester general chemistry course could be recognized and segregated from the main body of students. The chemistry department determined that the mathematics component of the SAT (formerly an acronym for Scholastic Aptitude Test, later Scholastic Assessment Test, and now simply SAT), the standardized general knowledge test that is widely used as a college entrance examination in the United States of America, could be used as a metric to predict student success in the first semester course of the general chemistry program. It was observed that 70% of students who failed to achieve a grade of C or higher, and were therefore not allowed to continue on to the second semester of the two-semester general chemistry program, had mathematics SAT (MSAT) scores lower than 580 (for reference, the MSAT scores are normalize to range from 200 to 800). The author's home institution decided to develop an alternative program for these lower scoring students, expanding the first semester of the traditional two-semester general chemistry program into a one year, two-semester course, and – with the second half of the traditional two-semester program – producing an alternative three-semester general chemistry program. Incoming first year students were then placed in either the standard two-semester program or the alternative three-semester program based on a 580 cutoff MSAT score. It is important to state here that while a student's individual MSAT score was in no way a good predictor of that student's final grade in the standard or alternative general chemistry program, the MSAT score could predict pass/fail, and the cutoff was used as a way to place students in the program where they would have a good chance of success. Students were given the metrics data, informed of the rationale for their placement, and then were allowed to opt-in or out of the standard and alternative programs.
While the three-semester alternative program had some success (approximately 50% of the students placed in the program passed the first and second part of three semester program and were allowed to progress to the third semester – despite the metric that indicated that they would not succeed in the equivalent first semester of the standard, two-semester, general chemistry program), we wanted to see if this lower performing segment of students would perform better with alternative teaching methods. The premise was that these underperforming students tend to not do well in traditional teaching settings, but could possibly do well enough in alternative teaching methods that were aligned with our understanding of learning. We already observed a student performance boost when the traditional print-textbook was replaced with Flash-based animations and simulations (Gregorius, 2011, 2013). We wanted to see if a flipped classroom teaching method would further improve student performance and pass rates.
The study was conducted every fall semester over a four-year period on two sections of the first course of the three-semester alternative college-level general chemistry program. On average, each section would begin with 33 students with an additional 12 students per section opting into the three-semester program after doing poorly in the first exam in the two-semester program (an option which we allowed). The statistics on MSAT and Verbal and Critical Reading SAT (SATL) scores for students finishing the program and getting a grade over the four-year study are shown in Table 1. A breakdown of the MSAT and SATL scores by year in the study is shown in Table 2. It can be seen that the student aptitude and preparedness as measured by the SAT is consistent from year to year. We felt confident that student performance across the years of the study could be compared.
Students finishing the program | ||||
---|---|---|---|---|
Starting in the program | Transferring in | |||
MSAT | SATL | MSAT | SATL | |
Average | 502 | 509 | 571 | 536 |
Median | 505 | 510 | 580 | 550 |
Std dev | 50 | 61 | 59 | 72 |
High value | 580 | 750 | 670 | 710 |
Low value | 300 | 330 | 350 | 390 |
Students finishing the program | ||||||||
---|---|---|---|---|---|---|---|---|
Starting in the program | Transferring in | |||||||
MSAT | SATL | MSAT | SATL | |||||
Year | Average | Median | Average | Median | Average | Median | Average | Median |
1 | 504 | 510 | 515 | 500 | 574 | 590 | 521 | 550 |
2 | 502 | 500 | 514 | 520 | 526 | 550 | 590 | 610 |
3 | 503 | 500 | 512 | 510 | 584 | 590 | 536 | 545 |
4 | 496 | 500 | 507 | 510 | 573 | 570 | 520 | 525 |
The course in this study covered the first five chapters of a typical college/university general chemistry textbook: (1) an introduction to matter, units and measurements, and dimensional analysis, (2) the basics of atoms, ions, ionic and molecular compounds, and binary inorganic nomenclature, (3) fundamentals of empirical and molecular formulas, chemical equations, and stoichiometry, (4) a treatment of aqueous solutions and reactions in aqueous solutions, including precipitation, acid/base, and oxidation reactions, as well as the concepts of concentration and the molarity concentration unit, and (5) a treatment of thermochemistry: first law of thermodynamics, enthalpy, calorimetry (constant volume and constant pressure), Hess's law, and enthalpies of formation.
There were four summative assessments: three midterm exams and a final exam. The first midterm exam covered the first two topics described previously. The second midterm exam covered the third topic and the first half of the fourth, covering aqueous solutions and reactions but not molarity. The third midterm exam covered the remainder of the fourth topic and the final topic, thermochemistry. The final exam was comprehensive. The midterm and final exams were structured so as to capture the true knowledge of the students. A multiple-choice section, similar in structure to the American Chemical Society (ACS) paired-questions exam (ACS, 2005) had a balance of concept-based and application-based questions. We also tried to have to have both “easy” and “more involved” question over the same topic. An easy concept question on writing and balancing chemical equations might take the form excerpted in Fig. 1. A more involved variant of a concept question over the same topic might take the form shown in Fig. 2. On the other hand, an easy applied question on the same topic might take the form: “When the chemical equation for the combustion of C3H8O is balanced, the sum of all the coefficients is …” We also included an open response section. Students were required to show their work on calculations or write essays/draw figures in response to these types of questions. 50% of the midterm exam score came from this open response section.
The traditional/standard two-semester general chemistry program was structured such that students who did poorly in the first midterm exam in that course could transfer to the alternative three-semester program after the first exam. The first exam performance in the two-semester program was by far the best indicator of potential success or failure in the general chemistry program. Our metrics indicated that ninety percent of students who failed the first midterm exam in the standard two-semester program were likely to withdraw or get a grade of D or lower in that course. Allowing these students, who passed through the 580 MSAT cut-off filter, the option of transferring to the slower-paced three-semester program offered a greater chance for success. This meant that of the three midterm assessments in the three-semester program, the first was given to students whose MSAT scores were below the 580 cut-off and who opted to start in the alternative program, while the second and third midterms included students who were above the cut-off or who opted out of the alternative program but did poorly in the first exam of the two-semester program.
Hoping to get a clearer picture of student performance relative to the teaching method, we decided to further differentiate the second and third midterm assessment by reverting to a traditional lecture and textbook for the third topic coverage. As such, we had two semesters worth of student performance data in a traditional teaching method in the three-semester program, two semesters worth of data of students below the MSAT cut-off in a flipped classroom teaching method for the first topic and midterm exams, two semesters of data for a flipped classroom teaching method with a mix of students below the MSAT cut-off and students above the MSAT cut-off but who performed poorly in their first exam in the standard two-semester program taught in a traditional lecture and textbook system and who then transferred to the three-semester alternative program, and we had assessment data on the same mix of students taught in the traditional lecture and textbook method for the third topic in the alternative program.
The traditional lecture teaching method followed the following protocol: a lecture, direct instruction over the current topic, following the sequence in the textbook (Brown et al., 2015) used by the entire department, prescription of exercises taken from the textbook or the online, electronic assignment system allied with the textbook, in-class group work activity after a block of content had been covered and serving as a formative assessment, a quiz serving as a short summative assessment, and when the entire section had been covered in this way, a midterm exam, as described previously, covering the section topic.
The flipped classroom teaching method followed the following protocol: assignment of an animation module covering the upcoming topic, a worksheet, which served as a learning guide, was coupled to the animations. Screenshots of a typical animation module are provided in Fig. 3, and a more thorough treatment of the design principles of the animations is provided elsewhere (Gregorius, 2008, 2010, 2013; Gregorius et al., 2010b). An excerpt of the accompanying worksheet/study guide is provided in Fig. 4. Students were expected to use the worksheet to ensure the proper development of content knowledge. In the classroom, at least two days after the release of the appropriate animations and worksheets, a series of pre-developed questions, similar in coverage and focus to the worksheets, were given in sequence. This “questions document” served as a structured treatment of the content without the process devolving into direct instruction or a lecture. Students answered the series of questions – with the instructor eschewing actually providing the answers to the questions but guiding the critique of the answers provided by students. The answers that the students agreed upon would be noted on the questions document, and later, when the entire questions document was completed by the students, the questions document would be published through the class's web course tool and serve as class notes and a statement of content understanding. This procedure was viewed as allowing students to take ownership of the content knowledge, and also served to prod the students to go through the animations and worksheet prior to the classroom activity. After a block of content was completed, similar to the traditional teaching method described previously, group work activities, a quiz, and a midterm exam followed.
The students' time allocation for the course was expected to be nearly equivalent for the traditional and flipped classrooms. While more study time would have to be put in to peruse the animations in the flipped classroom, this would be partially offset by the absence of assigned homework or graded assignments. This expectation was supported by formal surveys and informal student reports. There was no significant difference in the hours spent per week studying for the course between the two teaching methods. The average and median study time per week in the flipped classroom was 2.2 h and 2 h, respectively, while it was 1.9 h and 2 h in the traditional setting. The content coverage and class hours spent was identical in both settings.
Student performance over two semesters in both the traditional and flipped classroom settings was monitored. The final grades and midterm exam performances are reported here and serve as indicators of the potential effectiveness of the flipped classroom teaching method.
The final overall grade distribution, including students who withdrew from the course and received a grade of W, is shown in Fig. 5a. What stands out is the percent of students who withdrew from the course in the traditional setting: 52% of 208 students versus 22% of 154 students in the equivalent flipped classroom setting. At first glance, with a higher percentage of students in the flipped classroom setting completing the course, it would seem that the students in the flipped classroom outperformed their counterparts in the traditional setting.
Fig. 5 Overall performance: traditional vs. flipped. (a) All students and (b) students with recorded grades only. |
If our objective were to have as many students pass the first semester of the alternative three-semester course, than it would seem that a flipped classroom was a more effective teaching method than the traditional approach. Informal interviews and polling of students in the flipped classroom pointed out that students felt that they had “a good chance of making it”, that the material “made more sense”, that they “could always go back” and review the material – all of which contributed to students deciding to stay in the course. Other investigators have reported similar findings on student preference for the flipped classroom teaching method (Fitzgerald and Li, 2015; Seery, 2015; Weaver, 2015), and of students having a sense of confidence in their grasp of the material when presented in this way (Schultz et al., 2014; Hibbard et al., 2016; Shattuck, 2016).
However, if the objective was to enhance the learning of students (as measured by performance in exams), we might look to the performance of only the students who remained in the course and received a grade. If the students who eventually withdrew were taken out of the analysis, a somewhat different picture appears. The grade distribution for just the students who took the final exams and had a grade recorded is shown in Fig. 5b. There appears to be no significant difference in the percent distribution of students through the range of letter grades. It would seem that the strongest effect of the flipped classroom intervention is in the percent of students who successfully completed the course, rather than a better performance for those who did complete the course. As such, we wanted to delve deeper into the student performance of those who did complete the course in the hope of getting a clearer picture of student performance under the two teaching methods.
The performance in the first midterm exam for only the students who completed the course in both the traditional and flipped classroom teaching methods is summarized in Fig. 6a. There were n = 84 in the combined 4 sections over two semesters of the traditional teaching method, and n = 85 in the flipped classroom teaching method. The average and median scores for the traditional settings were 68 and 65, respectively, and 64 and 66 for the flipped classroom. It would seem, from the median and average scores, that there is no difference in student performance when subjected to the two different teaching methods. However, it can be seen from the graph of grade distributions that there was a greater percentage of students in the extremes in the flipped classroom setting. There were 4% of students getting A's in the traditional setting, while there were 11% in the flipped classroom. On the other hand, there were 12% getting D's in the traditional versus 21% in the flipped. This seems to point out that within the group of underprepared students (those who came into their first year of college with a MSAT of 580 or lower), there is a small subgroup that might thrive in a flipped classroom setting, but there is also a small subgroup that would perform worse when compared to those in the traditional teaching method.
A similar pattern can be seen for the student performance in the second midterm exam, when students transferring from the two-semester program had been incorporated into the study groups. This is summarized in Fig. 6b. There were n = 100 in the combined 4 sections over two semesters of traditional teaching method, and n = 119 in the flipped classroom teaching method. Average and median scores for the traditional settings were 62 and 63, respectively, and 63 and 65 for the flipped classroom. Again, it would seem, from the median and average scores, that there is no difference in student performance when subjected to the two different teaching methods. However, the same trend of students in the flipped classroom either doing better or worse than their counterparts in the traditional setting can be seen. 1% of students in the traditional setting received A's, while 8% did so in the flipped classroom. On the other hand, there were 18% getting F's in the traditional versus 26% in the flipped classroom setting.
This trend of the flipped classroom being more effective for the upper bracket and worse for the lower bracket of students in this study can be further highlighted when the transfer student performance was separated from the overall student performance in this study. Fig. 7 separates the performance in the second midterm exam of students above and below the 580 MSAT cut-off; Fig. 7a shows the data for students who started the semester in the three-semester program, while Fig. 7b shows that of students who transferred into the alternative program after doing poorly in the first exam of the two-semester program. It can be seen from these graphs that, again, there is a wider spread of student performance in a flipped classroom setting, with more students performing better and worse than their traditional counterparts, and that this difference in overall performance is independent of whether the students started the semester in the three-semester, flipped classroom program or not.
The third midterm exam was over content that was handled in a traditional teaching system for both groups in this study; the expectation was that there would be no difference in student performance between the two groups. However, it can be clearly seen in the graph shown in Fig. 8 that the students who had developed content in the flipped classroom teaching method performed poorer than their counterparts in the traditional setting when the teaching method reverted from flipped classroom to traditional teaching methods. Splitting the performance comparison into students who started in the alternative program (Fig. 8b) from students who transferred in after the first exam (Fig. 8c) showed that the poor performance for students who started in a flipped classroom setting and reverted to a traditional lecture setting for the third content and exam was independent of whether they started in the alternative program or transferred in midway through the semester. It seems that after a student had been studying in a flipped classroom setting for a certain period, going back to a traditional teaching/studying method had an adverse effect on the student's performance.
Fig. 8 Performance in the third midterms. For (a) all students, (b) students who started in the alternative program, and (c) students who opted-in. |
When the performance of only the students who completed the course was analyzed, it appeared that there were subgroups of students who either performed better or worse in the flipped classroom setting when compared to equivalent students in the traditional teaching method. This seemed to indicate that while students may get a sense of confidence in their grasp of the content in a flipped classroom setting, enough to put effort into the course and complete it, this sense did not readily translate to increased learning (as measured by performance in exams) for some of these students. The exam performance average and median scores for students who completed the flipped classroom course and for those who completed the traditional lecture-based course were similar enough to suggest that the level of learning for both groups in the study was comparable.
However, the average and median scores mask the difference in the spread or distribution of letter grades. The shape of the distribution curves were sufficiently different that it could be noted that there were more higher scoring as well as low scoring students in the flipped classroom. This could mean that a flipped classroom teaching method might have been too much for a small subset of students, who worked to stay in the course but could not perform well. We may need to be aware that not all students will thrive in this teaching method. On the other hand, this could mean that since the retention rate is higher in the flipped classroom setting, and considering that the students in this study are generally underprepared for college-level study, the shape of the letter grade distribution in the flipped classroom might have been skewed toward lower letter grades with the inclusion of more underperforming students. In this case, the flipped classroom student performance curve might actually represent higher learning outcomes. Further study needs to be made in order to determine if – higher retention rates not withstanding – an animation-based flipped classroom allows for higher learning outcomes, and/or the development of better understanding of particular topics.
Another important finding of this study was that while these underprepared students could readily adjust to a no-lecture, animation-based flipped classroom teaching method, after a certain amount of exposure to this method – the length of one content topic or chapter – reverting to a traditional teaching method was shown to have an adverse effect on student performance. This seems to indicate that, for these types of students, using multiple and very different teaching methods, especially if the novel teaching method is used for a prolonged period before switching to a traditional teaching method, would have a deleterious effect on student performance.
Contrary to the preconception among some instructors that the flipped classroom teaching method will not be effective for underprepared students, this study seems to indicate that even underprepared students, the bottom third of incoming first-year college students, will perform well in the flipped classroom. While there will be some students who will be adversely affected by, and have difficulty adjusting to, this novel teaching method, this effect is offset by the percentage of students who persist in the program, and the percentage of students who perform better in the flipped classroom setting despite coming in underprepared.
While the teaching method used here had a significant impact on student retention even among underprepared students, there is concern that, based on the performance of students who did complete the course, learning was not particularly enhanced for all students. Some students thrived in the teaching method while others fared worse than their counterparts in a traditional, lecture-based course. It may be necessary to temper our expectations of what this teaching method can allow for our students.
It is important to note that, should instructors choose to adopt this teaching system, there is data here suggesting that switching back to a traditional lecture-based teaching method after using the flipped classroom method for at least one unit or book chapter can have an adverse effect on student performance in later chapters. Rather than using the method for specific content or switch back and forth between different teaching methods, it might be better to develop an entire semester as a flipped classroom.
This journal is © The Royal Society of Chemistry 2017 |