Fusing a reversed and informal learning scheme and space: student perceptions of active learning in physical chemistry

Abstract

Physical chemistry students often have negative perceptions and low expectations for success in physical chemistry, attitudes that likely affect their performance in the course. Despite the results of several studies indicating increased positive perception of physical chemistry when active learning strategies are used, a recent survey of faculty in the U.S. revealed the continued prevalence of instructor-centered approaches in physical chemistry. In order to reveal a deeper understanding of student experiences in an active learning physical chemistry course, we present a phenomenological study of students’ perceptions of physical chemistry when the course is completely redesigned using active learning strategies. Using the flipped classroom, an active learning space, cooperative learning, and alternative assessments, we emphasized fundamental concepts and encouraged students to take responsibility for their learning. Based on open-ended surveys and interviews with students, we found that students struggled with the transition, but had some significant positive perceptions of the approach. This is in agreement with previous studies of physical chemistry courses in which cooperative learning was the focus. As part of a larger study of the effectiveness of this course redesign, we show how students perceive the effectiveness of these strategies and how they react to them. In addition, we discuss the implications of these findings for the active learning physical chemistry classroom.

---

Introduction

The topics covered in a Physical Chemistry course are fundamental to the understanding of chemistry, physics, engineering and many other related fields. Due to prerequisite skills acquired in calculus and physics courses, Physical Chemistry is often integrated into the curriculum as an upper-level undergraduate course for students in the major (Committee on Professional Training, 2015). It is well known by practitioners, but also demonstrated by survey results and interviews, that chemistry students often enter Physical Chemistry with negative perceptions and low expectations for success (Nicoll and Francisco, 2001; Sözbilir, 2004). Specifically, lecturers and students alike have identified the abstract nature and mathematical content of thermodynamics as a source of student difficulties (Sözbilir, 2004). In fact, many alternative conceptions and student difficulty with math transfer have been identified in the chemical education literature (Bain et al., 2014). Much effort has been placed on identifying factors of success for physical chemistry students (Derrick and Derrick, 2002; Hahn and Polik, 2004; Nicoll and Francisco, 2001), but less emphasis has been placed on student perceptions of the course (Bain et al., 2014). As early as 1989 (Carter and Brickhouse, 1989) and more recently (Xu et al., 2013) student attitudes and self-concept have been identified as predictors of success in introductory chemistry courses. Logically, practitioners may consider that increasing student perceptions of physical chemistry may result in increased performance.

The term active learning refers to a myriad of constructivist course designs that can range in the degree and execution of strategies such as group problem solving, the use of classroom response systems, or worksheets completed during class (Freeman et al., 2014). According to Freeman, et al. (2014) “active learning engages students in the process of learning through activities and/or discussion in class, as opposed to passively listening to an expert. It emphasizes higher order thinking and often involves group work.” Several studies show increased positive perception of physical chemistry when active learning strategies are incorporated (Becker et al., 2015; Hinde and Kovac, 2001; Partanen, 2016). Conversely, other studies show that students experience discomfort in the transition (Pentecost and James, 2000). In any case, there is an abundance of evidence in support of the effectiveness of any version of active learning in STEM courses in terms of performance (Freeman et al., 2014).

Despite the evidence in support of active learning, a recent survey revealed that a majority of physical chemistry faculty (79% of those surveyed) continue to teach using instructor-centered approaches like lecture to deliver content (Fox and Roehrig, 2015). This was even more common among faculty at large institutions similar to the institution at which this study took place. Of the few notable exceptions reported in the literature, faculty, for example, modified the lecture to include frequent breaks and collaborative problem-solving (Partanen, 2016) or completely redesigned the course using cooperative learning (Pentecost and James, 2000; Towns and Grant, 1997) or process-oriented guided inquiry (Becker et al., 2015).

Considering that student perceptions of a course can affect their emotions, motivation, and thus learning and performance, we decided to embark on the investigation of student perceptions of a physical chemistry course taught using active learning strategies. The design of the course was rooted in two currently emerging popular pedagogical approaches in chemical education: the flipped classroom (Seery, 2015) and active learning spaces inspired by the SCALE-UP learning environments introduced at North Carolina State University (Beichner et al., 2007; Weaver and Sturtevant, 2015). Use of these approaches offered significant opportunity for less formal assessment than the traditional chapter exam, so exams were eliminated in an effort to reduce test anxiety and increase focus on deep conceptual understanding rather than rote learning. The effect of implementing these active learning strategies was studied using a phenomenological approach. As part of a larger study of the effect of using active learning strategies in physical chemistry, the research question that guided this part of the study was: What are student perceptions of active learning in a physical chemistry course?

Student perceptions of Physical Chemistry

Two noteworthy studies have explored student perceptions of physical chemistry in a traditionally taught course. Nicoll and Francisco (2001) identified students’ anticipation of the difficulty of the course and the considerable amount of time they would be required to devote to studying. Later, Sözbilir (2004) investigated perceived difficulties in more detail. The factors influencing students’ perceptions were categorized as student, course, and staff. The largest contribution to the student category was lack of student motivation or interest in the material. On the other hand, the largest contribution to the staff category was teacher-centered learning. This leads us to think that a move away from teacher-centered learning might increase student motivation and interest. In fact, students’ suggestions for reducing these difficulties included faculty promotion of group work and discussions. Students attributed difficulties associated with the course to abstract concepts, overloading of course content, and superficial understanding of concepts among other factors. As shown next, these factors have been mitigated in other courses using various active learning strategies.

In 2000, Pentecost and James used a flipped classroom approach facilitated by instructor-provided guided reading materials in order to give students more responsibility for learning. Interviews with their students over three implementations of the course revealed mixed emotions about the change in teaching style. Their results indicated that while students may value the goals of active learning strategies in theory, the drastic change might initially lead to negative perceptions.

Other studies revealed largely positive student perceptions (Hinde and Kovac, 2001; Partanen, 2016). These results may lend more encouragement to faculty to incorporate active learning strategies into upper-level chemistry courses. Introduction of different degrees of cooperative learning into two physical chemistry courses received positive student feedback (Hinde and Kovac, 2001). Students said that the cooperative learning activities led to a deeper understanding of concepts. These students felt motivated by the flexibility of the course structure, the pressure to contribute to group work, and the experience of relating to their peers (i.e. realizing that others have the same conceptual difficulties). More recently, Partanen (2016) incorporated student-centered strategies including exam “cheat sheets” and interactive lectures with breaks and activating tasks like example problems. Her students also responded well to the incorporation of these strategies, especially group work since they felt more comfortable communicating their misconceptions to their peers. An especially interesting result of her interviews with students was in regards to the origins of the negative perceptions coming into the course. Their negative impressions were triggered by students who had taken the course previously and been unsuccessful in passing. This indicates that negative perceptions can be mitigated early on if students see that passing the class is an obtainable reality.

Taken together, the results of these studies suggest that with thoughtful planning, active learning might increase positive affect and student perception of physical chemistry, thus increasing student performance, and, more importantly, conceptual understanding of this fundamental field of chemistry.

Methods

Research design

Considering the emphasis of the research question on student experiences, we decided to use a phenomenological research design in order to elicit rich descriptions of student perspectives and be able to understand more deeply their experience in an upper-level course that uses active learning. Phenomenological research studies are designed to describe the common lived experiences of a group of individuals experiencing the same phenomenon. They describe the “what” and the “how” of the experience largely through interviews (Moustakas, 1994). However, phenomenology has been said to lie on the line between qualitative and quantitative research (Creswell and Poth, 2017) so data such as observations can be used to help describe the experience as well.

Exploring student perspectives of physical chemistry is not in itself a novel research endeavor, but in this course design, students were expected to take most of the responsibility for their learning with the instructors acting as coaches. That level of active learning has not yet been described in the literature in terms of student experiences. This information could have implications for upper-level chemistry instructors who expect their students to have the skills necessary to be successful in this type of learning environment and inform the practitioner about how to implement active learning strategies more effectively.

Setting and participants

This study was conducted at a public research university with a total undergraduate enrollment of roughly 56 000. Sixty-three students enrolled in Physical Chemistry I in the Fall 2016 semester and agreed to participate in the study. In order to participate in the study, students had to be present on the first day of class to receive the Explanation of Research with the course syllabus and to take a pre-instruction assessment. The Explanation of Research described the goals and procedures of the research and explained that the students could choose not to participate without detriment to their course grade. This protocol (#SBE-16-12307) was approved by the university's Institutional Review Board (IRB) after an expedited review. Their class standing, academic program, gender, and race/ethnicity distribution are presented in Fig. 1.

Fig. 1 Class demographic data.

The course was an upper-level Physical Chemistry I course which focuses heavily on chemical thermodynamics and basic kinetics and is required for all chemistry majors to take in most university chemistry programs. The class met four days a week for 50 minutes. The corresponding author served as the professor for the course. He had taught this course six times before, the graduate chemical thermodynamics course eight times and the undergraduate physical chemistry laboratory eleven times. He facilitated the whole group discussions, chose homework and class activity problems, and wrote the questions for the conceptual assessments. The primary author acted as the teaching assistant, providing support to students during group activities and facilitating class on occasion.

Instructional approach

Fig. 2 describes schematically the instructional approach used based on the implications of findings from previously discussed studies of factors that influence student success in physical chemistry. Two main strategies served as the foundation of the course structure: the reversed learning scheme (or flipped classroom) and the informal learning space, (or active learning space). Because of the substantial opportunity for assessment using these approaches, we decided to eliminate the traditional chapter and final exams. Instead, students were evaluated solely on weekly homework assignments, in-class group activities, and participation in class discussions. Students received weekly feedback on homework problems and daily feedback via posting of correct answers to in-class activities on the online learning management system. In addition, ungraded pre-instruction, midterm semester, and post-instruction assessments were administered. These assessments were administered as part of the evaluation of the effect of this course structure on student conceptual understanding, but interview findings revealed that students might have used these ungraded assessments as a measure of their own growth.

Fig. 2 Schematic of instructional approach. The reversed and informal learning scheme and space (RAILLS) served as the foundation. Purple labels are alternative assessments used throughout the semester. SoTL is included in the schematic as the structure was constantly (and is continuously) being evaluated and modified.

The first main strategy used was an adaptation of the flipped classroom approach in order to establish a culture of inquiry before students arrived to class. Whereas in most flipped classrooms, students watch video lectures prior to class (Seery, 2015), we asked our students to read one chapter and complete two comprehensive questions from the text. These homework assignments were graded based on correctness with partial credit by the instructor. This was similar to the approach taken by Pentecost and James (2000), but by assigning comprehensive problems in place of a guided reading activity. In order to ease the transition, we dedicated the first class meeting to modeling how one would extract concepts from the text. The instructor read main headings and subheadings from the text and engaged the class in a questioning-predicting exercise for the first half of class. Second, although the furnishings associated with current “active learning” classrooms based on the SCALE-UP classroom design (Beichner et al., 2007) were not available for us to use with this class size, we considered the space an “active learning space”. Students were able to move their desks, work independently or in groups, and utilize any technology they had available.

Class time most often began with a whole group discussion and ended with a small group activity. The instructor facilitated the whole group discussion, emphasizing main ideas from the chapter and key points that may often be overlooked or misunderstood. The whole group discussion was purposefully utilized as an attempt to correct alternative conceptions students usually have in physical chemistry (Bain et al., 2014). We thought directly addressing some of these concepts (i.e. equilibrium) may help correct misconceptions. Small groups (4–5 students) were alternately chosen by the students or assigned by the instructors based on performance on the homework for that week.

Class activities were tentatively assigned at the beginning of the semester (students received a tentative schedule with the syllabus with homework and class activity assignments). Following whole group discussion, considering the concepts covered during the whole group discussion and the time remaining in class, the class activity was assigned. Most commonly, the activity was a problem directly from the text. For example, on the day that temperature and the zeroth law of thermodynamics were discussed, very little time was left (about five minutes) to answer the two assigned conceptual problems for that day. Based on this, only one question (Can you think of any property of a living system that could be used as a thermometer?) was assigned as an individual exit slip type of activity which students completed in the last 2–3 minutes of class. When more time was left, students might be posed with a more involved problem. For example, when covering equilibrium constants, one class activity was as follows:

The equilibrium vapor pressure of water over K₄Fe(CN)₆·3H₂O(s) plus K₄Fe(CN)₆(s) has been reported as 10.0 Torr at 25.0 °C and 7.20 Torr at 20.0 °C.

(a) Compute K at 20.0 °C and 25.0 °C for the reaction

K₄Fe(CN)₆·3H₂O(s) ⇌ K₄Fe(CN)₆(s) + 3H₂O(g)

(b) Compute at 25.0 °C for the reaction.

(c) Compute .

(d) Compute .

Class activities were to be completed by the end of the class period during which time the instructor and TA would circulate to answer questions and guide students. The assignments were graded based on correctness with partial credit by the TA.

In order to monitor student progress, a pre-instruction, midterm semester, and post-instruction assessment consisting of conceptual questions written by the instructor were administered. The pre-instruction assessment was administered on the first day, the midterm halfway through the semester, and the post-instruction assessment on the day that the university schedules a final exam for the course. The results of these assessments were used to study student conceptual change as a part of a larger study, but they were not used to provide feedback to the students or the instructors and they were not graded assessments. Homework and class activities served those purposes.

Data collection

In order to assess student engagement, we monitored attendance using class activities and asked our colleagues to observe the class. The Behavioral Engagement Related to Instruction Protocol (Lane and Harris, 2015) was used since this was a relatively large enrollment class. Graduate students interested in physics and chemistry education were asked to observe the class. Observers were provided with a list of engaged and disengaged behaviors as described by Lane and Harris (2015). They were asked to enter the room inconspicuously and choose ten students within their range of view to observe. In two-minute increments, they counted the number of students in that group exhibiting behaviors on the “engaged” list and took notes about student activities and behaviors during the class period. In addition, they took notes about what topics were being covered, what instructional strategies were being used, and where the students were sitting in the room relative to the instructor.

Following the post-instruction assessment, students were asked to respond to three demographic and one open-ended question: What did you like the most about this teaching style? The least? Please be specific. Students responded to the survey question using Socrative (MasteryConnect, 2018), a free classroom response system that students can access using their smart device or computer. Instructions for accessing the survey were included with the post-instruction assessment.

Semi-structured individual student interviews were conducted in the semester following completion of the course. All students who participated in the study were contacted via e-mail and asked to volunteer to participate in an interview. Six students volunteered to discuss their perceptions of the course and some thermodynamics concepts with the first author. Interviews were conducted with only the participant and the first author and lasted 30–45 minutes. The semi-structured interviews began with a question about the students’ general perceptions of the course. This was followed up by asking students to compare their experience in Physical Chemistry 1 to another traditionally taught course. Students were also asked about the dynamics between group members during class activities and whether they understood any specific topic better at the end of the course as a result of something that occurred during the semester. These questions were adapted from those used by Towns and Grant (1997) when they incorporated cooperative learning into their Physical Chemistry course. The complete interview protocol is presented in Appendix 1.

Data analysis

Sixty students responded to the survey and their responses to the open-ended survey question were coded by inductive content analysis (Elo and Kyngas, 2008). After choosing the unit of analysis (in this case, survey responses) the next step in inductive content analysis is for the researcher to become “immersed” in the data. This involves reading through the data several times in order to become familiar with it. Next, the researcher organizes the qualitative data using open coding and by creating categories. The categories are grouped with similar data until all of the codes contained in one heading are similar and “belong” to that group. Thus, themes arose from student responses rather than from any specific theoretical framework. Four major themes were identified and aligned fairly well with the strategies incorporated into the course redesign: homework assignments/flipped classroom, whole group discussions, group work, and alternative assessments. Notes on how comments were coded and sample responses for each code are provided in Table 1.

Table 1 Themes identified in survey responses. Similar themes were identified and analyzed in interview transcripts

Theme	Note	Frequency	Sample response
Alternative assessment	Students were not evaluated using traditional quizzes, tests or exams. This code was used when students referred to not having tests, exams or having their “only grades” based on other assessments.	19	(Negative example) I did not like having the only grades being homework and classwork because if you don’t understand the concepts on your own you don’t do well on them. Having no tests also minimized the motivation for me to learn.
Discussion	The beginning of most class periods consisted of a whole group discussion. This code was used when students referred to the discussions explicitly or to “lectures.”	18	(Positive example) I enjoyed how interactive the lectures were. I found it helpful.
Flipped classroom	Students were asked to read one chapter per week and complete a homework assignment consisting of two problems, usually assigned from the textbook. This code was used if students referred explicitly to the flipped classroom, homework or learning material on their own.	17	(Negative example)
			It's hard to complete the homework assignments ahead of the lectures.
			OR
			Teaching one self for this class is challenging.
Group work	Following the whole group discussion, students would either be asked to choose or were assigned groups to work with. If students referred explicitly to group work, this code was used.	8	(Positive example) The class assignments I liked, we were able to work in groups and it helped me learn the material from other people's perspective.

---

Interviews were audio recorded with the participants’ verbal permission, transcribed verbatim by the first author, and analysed according to the four major themes that arose from the survey responses.

Results and discussion

Attendance

Since high attendance has been correlated with high performance (Credé et al., 2010) and may give insight to student engagement, our first measure of student engagement with the course was attendance. It is important to note that, while attendance was not technically required, class activities accounted for 30% of the final grade. These activities could not be made up in the event of an absence, providing some extrinsic motivation for students to attend class. However, we did have a policy, which allowed only about 90% of the class activities to be graded so that any unanticipated absences or unusually poor performance on assignments would not be detrimental to the course grade. Using class activity participation as a measure of attendance, we determined that students attended, on average, 85% of scheduled classes. We have no available records of attendance for this course prior to Fall 2016, but according to Credé and co-workers (2010), this would be considered a high rate of attendance, indicating high student motivation, at least to attend class. A second interesting observation is that the retention rate from the first day of class to the last day of class was 70%, a good indication of persistence. In addition, there were no withdrawals or D's. Only one student received an F because they inexplicably stopped attending class early in the semester.

Observations

Four representative observations are displayed in Fig. 3 with the date of the observation in the lower right corner of each plot of student engagement versus time. Important to remember is that each plot represents the engagement of a group of 10 students in the class. Student engagement is reported as a fraction since there were some cases in which groups of 11 were observed if a student came in late and joined a group already being observed. Points representing engagement during an activity not related to course material (i.e. grading policies or moving into groups) are not shown.

Fig. 3 Observation data showing number of students engaged during every 2 minute interval.

As shown in the plots, with only three exceptions, at least 6/10 students in the observed groups were engaged at any given two-minute interval. Further analysis reveals that average engagement was 0.75 or above. These values are reported for each class period on the respective plots. It is especially interesting to compare the top two plots, which represent the typical class in which half was dedicated to whole group discussion and half was dedicated to group work, with the bottom two in which the majority of the class time is spent on only one activity. Surprisingly, engagement was slightly higher on average in the latter case. Specifically, the third observation was done during a class in which we asked the students to perform a “lab”. They were provided with cooling curves from a cryoscopic determination of molar mass experiment and asked to calculate the molar mass of an unknown and to identify it based on their results (they were provided with a list of possible substances).

Exploration of some specific points reveals interesting information. First, it is not surprising to see engagement alternate and then decrease toward the end of the discussion in the first observation (Bunce et al., 2010), but engagement is increased immediately upon starting the class activity. In the second observation, engagement drops around 12 minutes into class. At 18 minutes, engagement had not increased significantly so the instructor asked a question to help stimulate the discussion. During the group activity, engagement was relatively low at the beginning (0.6). The observer noted that three students were talking about something unrelated and one was using their phone. This is simply attributed to the transition to the new activity since engagement increases for most of the rest of the activity. The exception occurs around 6 minutes before the end of class when the observer noted that students had begun to do other work and pack up either indicating that they were finished with the problem or relying on a group member to finish.

Survey responses

After the post-instruction assessment was administered, we asked students to respond to one open-ended survey question: Sixty students responded to the question. Several themes including, but not limited to, those in Table 1, were identified in their responses. Other topics included the conceptual emphasis, textbook, instructor, participation requirement, and the informal learning environment. However, these topics were mentioned very infrequently, and thus are not included in the table. The complete set of themes is presented in Appendix 2. Here, the four most frequently discussed themes, alternative assessments, discussions, flipped classroom, and group work will be analyzed since they were also common themes addressed in the interviews and provided the framework for the structure of the course. All responses to the survey are reported verbatim.

The first topic was alternative assessments. Nineteen students discussed this topic on the surveys. Twelve of the nineteen responses referred to the alternative assessment approach as a positive attribute of the course. Eleven of those cited reduced stress as a reason for liking this approach. One wrote, “I enjoyed the class ideology, tests are stressful and do not assist in learning, only memorizing information.” and another, “I liked that I didn’t have to worry as much about test and could focus on actual learning. It was a nice change from having to memorize material.” Avoiding rote memorization and reducing test anxiety were precisely the goals we had for eliminating exams. This approach had other surprising and significant effects, though. Some students considered the fact that even without exams they still had to study. For example, one said, “I also liked no test even though each class was a “test” because if you didn't study ahead or read the material you would do poorly similar to an exam. It was a lot less stressful and I definitely learned a ton!” However, some students did not share this perspective, but had mixed emotions. Six of the nineteen responses referring to the alternative assessment approach indicated that students enjoyed not having tests, but four of those felt that this caused them to lack motivation, “It was nice not having tests but I think I would have put in more effort if there had been.” Only one student had a negative perception of the lack of tests and they cited motivation as the reasoning.

The second most common topic was the whole group discussion. Eighteen students mentioned this on the survey, seventeen of whom considered it a positive aspect. Six students explained that they like the informal nature of the lectures and the conversational, rather than traditional “lecture”, style. In fact, one student responded, “I felt like I was having a conversation about the topics rather than just being talked at for an hour.” Two students said they felt like the possibility that they would be asked to participate in the discussion motivated them to prepare for class each day. One student wrote, “The set-up of the class also helped since it was a group discussion and you would be randomly called on so you would have to be quick on your feet and actually know the material, which gave more incentive to learn the material.” Two students also mentioned that they enjoyed discussions because it gave them other students’ perspectives of the material. One wrote, “I enjoyed the participation from classmates as well, it helped me to pay more attention and to see where my mistakes and theirs’ lied.” These were common themes discussed during the interviews as well.

The third most discussed topic was also the strategy with the most negative perception. Seventeen students mentioned the flipped classroom approach with most emphasis on the difficulty of the homework. Sixteen students cited this as a reason they had a negative perception of this aspect. Most commonly, they expressed the belief that they were required to master concepts before they were taught in class: “…but I did not like the flipped classroom teaching style. We should not have been expected to do the homework without any instruction on the chapter.” and, “However, it was especially stressful to complete homework without being taught the material beforehand, and I believe it is not an efficient way for students to learn.” Two students mentioned the amount of time they had to devote to the homework outside of class. For example, “The homeworks required a lot of time, between reading and trying to understand the chapter and then trying to actually solve the problems it would take me at least 12–15 hours, sometimes more.” One student even said, “I feel like this entire course was essentially “figure it out on your own”, which is not what I’m paying tuition for.” which is a similar sentiment made by a student in Pentecost's (2000) study. One student had a neutral perception, recognizing the skills they developed by using this method, “I didn’t like how the math and practical problems I would usually have to figure out by myself through the readings and homework. Although in the long run I felt like it helped me more than traditional teaching methods because I would probably forget it all within a semester.” Only one student mentioned the flipped classroom in a positive light, “the homework builds knowledge before the subject is covered.”

Group work was less frequently discussed, but a very important component of the course and discussed at length in interviews. Eight students mentioned this component on the survey. Five of these considered group work a positive aspect and either expressed that they like to work with their peers or that they like getting their peers’ perspectives. One of these students added a contingency, though, “I enjoyed doing group work, because I work better with other people (when everyone participates and actually tries).” This was another student's reason for disliking group work, “The least aspect I liked about this course is the fact that when placed in group discussions some people do not put in the same amount of effort as others.” One student did not like “being forced into specific groups.”

Three other themes we identified were the conceptual emphasis in the course, the instructor, and the textbook. Seven students expressed that they particularly liked the conceptual emphasis, but one requested, “I wish we would go over more math based problem (sic) really helps with concepts.” Five students had strong negative opinions about the textbook, which, as a side note, was chosen by the instructor specifically because of its emphasis on concepts. Finally, all five mentions of the instructor were positive and mentioned his passion and enthusiasm for teaching.

Interviews

The purpose of the interviews was to explore the four major themes discussed in the surveys (alternative assessments, discussion, flipped classroom, and group work) in more detail. These discussions with students also revealed some interesting trends in student engagement and motivation throughout the course.

Alternative assessments. Many students cited reduced anxiety on the survey as a reason for liking the “no tests” approach. During the interviews, Student 6 agreed with this view. She expressed that she was “not very good at taking tests” and that while there was still pressure to complete homework assignments, she felt like she could focus more on learning and understanding, rather than memorizing and regurgitating when she did not have the pressure of an exam:

Student 6: I mean the fact that I…that class put absolutely no pressure on, I mean there was pressure to get the homework done because it was, like, graded, but at the same time there was no pressure in terms of memorization and regurgitation in the class. It's like he sat down and made sure that, hey, you know, this is how you do a problem. This is why it's like this in the book.

Student 4 explained that not having tests did not necessarily make the course any easier. When asked what she thought about not having exams, she said, “I liked it too, but like the homeworks, they were hard. It makes you put a lot of time on P. Chem.” These responses are interesting to compare with Student 3's response:

Student 3: Um…I really enjoyed the way the class was taught. I didn’t really like the no test method, though ‘cause as a student I feel like a test is a way of…pressures you to learn. And without tests you lose that stress and that pressure…that pressure to learn, learn the material so um…I don’t like the no test method. I thought like if the class was exactly the same, structured the same just with three tests throughout and a final, that probably would, I mean you’d learn a lot more P. Chem. 1.

On the other hand, Student 2 missed having exams because she felt like she could better track her progress with periodic formal assessments. She said, “I like having exams. It kind of, um, lets you track your progress and if you’re retaining anything.” Student 6 had an opposing argument, though:

Interviewer: So do you feel like taking the pre, mid and post kind of helped you reflect on what you had learned and see…?

Student 6: My progression in the class, most definitely. It definitely, um, opened up my eyes and it like got me excited. The fact that I was retaining the information and not just memorizing, regurgitation and, you know, and that's a thing where, ‘cause that's a thing where from my…my pretest, my midterm and then my posttest, like none of my answers are probably the same exact, word for word, this is what's going on in terms of things, you know. And I thought that was really exciting – actually going into a test and actually knowing, you know, what's going on in the problem and not just, you know, “Oh. This is what you said. I’m pretty sure this is what the answer is.”

All four students refer in some way to anxiety, stress or pressure to perform. However, Student 6 appreciated the reduced text anxiety because it allowed her to focus on learning. Student 4 still experienced some pressure due to the challenging nature of the homework assignment while Student 3 expressed feeling little pressure to “learn” the concepts because he never had to take a test or exam. Later in the interview, Student 3 was discussing strategies he and his classmates used to complete the homework assignments and said, “…like I did study a little bit, just not a lot. Not what the course demanded.” It became apparent that Student 3 held some previous perceptions of what a Physical Chemistry course should require in line with what Sözbilir (2004) found in his study. Interestingly, Student 3 received an A in the course regardless of his perceptions of the assessment method. According to Nicoll's (2001) findings, attitudes about physical chemistry do not play a significant role in student performance. However, we wonder whether Student 3 would have performed better on the post-instruction assessment had he had Student 6's perception of the lack of exams. These results highlight the different perspectives and responses students have to certain strategies. Student 6 appreciates the reduction of test anxiety which lets her focus on understanding, but can still track her growth using alternative assessments. On the other hand, Student 3 considers test anxiety a motivation to learn and does not initially consider working on homework assignments as “studying.”

Discussions. Two students on the surveys expressed that the discussions made them more accountable for knowing the concepts covered in class each day. Students 1 and 5 shared this view:

Student 1: Um…by far my favorite was when Dr. Hernandez would randomly call on people to answer questions. Because it forced…like I have to know what's going on right now and just be prepared.

Student 5 was more explicit about his underlying motivation during the discussions:

Student 5: Yeah and when he would call on people at random and put them on the spot. That would be the ultimate motivation – fear of public humiliation.

Interviewer: So that was motivating, not demotivating for you?

Student 5: Yes. ‘Cause you wouldn’t want that to happen again.

These comments agree well with some of the survey comments about the responsibility for knowing material and may account for the high engagement observed during discussion. Even when students had a negative perception of this pressure, they recognized its value. Student 3 explained that contributing to discussions was stressful, but he “liked” it. However, he felt that most students (not referring explicitly to himself) were not prepared, “so it didn’t work.” Interestingly, he blamed the characteristics of the students at this particular institution, but acknowledged the value of the strategy itself.

Homework/flipped classroom. The flipped classroom approach was the least popular topic on the surveys. Student 1 felt like the assigned homework problems were too complex.

Student 1 : I think it would have been better to have more questions that were a simpler level ‘cause the homework was a little over…like over my head sometimes. And so then I would just end up getting confused and copying something down and I don’t know if I necessarily learned as much as I should have from it…”

The fact that the material was not taught in class before students were required to complete the homework was the major reason for students’ low perceptions according to the surveys. In fact, during the interview, Student 2 said that she was skeptical “that I can, um, read a difficult material and understand it basically on my own.” This largely negative perception is not necessarily surprising. O’Flaherty and Phillips (2015) acknowledge the “significant minority” of courses in which students have negative perceptions of the flipped classroom. For example, students in an inverted statistics course were “unsettled” with the lack of classroom structure and had difficulties orienting to class activities (Strayer, 2012). Additionally, most research on flipped classroom in chemistry is in introductory courses (i.e. general and organic) (Seery, 2015). Physical chemistry is often associated with abstract concepts (Sözbilir, 2004), so student resistance to taking the responsibility for learning is not surprising. However, further analysis of the interview responses gives insight to overcoming this challenge.

All students interviewed shared some of their strategies for completing the homework. The approach we, as instructors, like to think all students take was initially described by Student 4. She explained that she would read the chapter before Friday so she could go to office hours to ask questions and then complete the homework before class on Monday. Although, this is not the approach she took from the beginning. She explained earlier in her interview that she “thought it's like the other homeworks for the other classes: you spend three-four hours and then you’re done. So I left it to the last minute.” After the first homework, she realized that this approach was not sustainable and devised a new plan. She explained that this new approach made her more responsible since the material was not taught in lecture beforehand. She also learned to be resourceful and said, “It made me really work.”

Student 6 also felt like she gained valuable study skills from learning using this approach. In fact, she stated that she was currently taking a different approach in her Calculus class than she might have before. She pointed out that since the homework in her Calculus class had a relatively low impact on her grade, she had little motivation to complete it.

Student 6: “Um…I would say that taking P. Chem. 1 actually helped me transition over into calculus and use that in order to study for…you know, my math, you know, tests and stuff like that because, sure, the homework is required but it's only a small fraction of, like, the grade so…”

More commonly, students expressed during the interviews that they read the homework assignment and then looked through the book or online specifically for information to help them answer the question rather than reading the chapter, trying to understand the concepts, and then approaching the homework. Student 5 discussed how he approached the homework:

Student 5: Read the question. Skim the chapter looking for a similar problem you could draw upon. If that didn’t work, type the question directly into Google. See if that gives you an answer, but the book we used was so obscure that that never worked, really. Then failing that would be looking at lecture notes from other P. Chem. classes, going through those quick, but then most of the time I could get some idea of what I needed to do…

Interestingly, continuing his discussion, he revealed that this search, while not always fruitful in terms of what he needed to complete the assignment, always left him with some understanding of various concepts he came across in his search.

On the other hand, while Student 3 had a similar approach, he discussed the fact that searching for specific information often left him with a shallow understanding of the concept. When asked whether he looked at the homework questions first then found the information needed to answer:

Student 3: Those specific questions, not the concepts, like I knew I had to find an equation specifically talking about, you know, I don’t know, cell potential at this exact, like, temperature. So I’m just going to find one dealing with cell potential at varying temperatures and that's it. That's all I care about. I didn’t care about how the temperature affected it or anything like that. I just had to get done, so that's what I was thinking.

Student 3 admittedly did not necessarily understand the concepts after completing the homework. Rather, in agreement with findings from other studies, he was able to succeed by virtue of his perceived mathematical abilities and logical thinking skills (Hahn and Polik, 2004; Nicoll and Francisco, 2001). Regardless of the fact that teaching approaches meant to give the student more responsibility for learning were employed, some students were still able to perform well without really obtaining a deep conceptual understanding of the material. In his study, Hahn posits that homework completion is a good indication of study skills, thus predicts performance. However, Student 3 admittedly did not reflect on concepts when completing homework assignments, but was still able to achieve an A in the course. His account gives more details about how homework is completed. It shows that even if a student completes homework assignments, they may not necessarily grasp chemical concepts depending on the strategies they use. These findings are in agreement with those of students in a quantum mechanics course in which alternative conceptions of concepts stemmed from non-productive strategies used by students when doing homework (Gardner and Bodner, 2007). These students, as well as ours, exhibited what Gardner refers to as the “problem-solving mindset”: their expectation is simply to solve problems, which influences their approaches to answering homework questions. Gardner's students and ours used similar strategies including scanning the book for equations and following worked example problems.

Group work. On the surveys, a handful of students complained that during group work, effort was not put forth equally by all group members. This was also a finding in Pentecost's (2000) study, but was in contrast to students in Partanen's (2016) study who actually felt like responsibility to their peers motivated them to prepare before class. During the interviews, Students 1, 2, and 5 shared the inequality complaint. For instance, Student 1 said, “…there was one week where half my group members came late, the other half didn’t know what was going on and I had to end up doing the whole assignment on my own, which was fine, but we would end up losing points because we wouldn’t have it completed because I couldn’t get everything done.” Student 2 said that she usually took charge in the group, but that “there would always be a couple people who just kind of linger, who don’t do anything.” Student 5 also mentioned, “…there would inevitably be about two people that wouldn’t know anything. They’re just sitting there and they’ll say yes to anything you suggest…” These three students felt like they were the strongest in the group, and thus had to carry the weight of the rest of their classmates. These student perceptions do not seem to be in agreement with observational data that shows high engagement during group activities. However, one observer noted the emergence of one student in each group as a leader. We acknowledge the possibility that even though it seemed as though their classmates were not engaged, they might simply not have been contributing in the way the leader expected. One observer even recorded 9/10 for one data point during group work, but then crossed it out and wrote 10/10. He noted that the student seemed to be disengaged, but when a group member asked them a question, they answered in such a way that indicated they had, in fact, been engaged during the small group's discussion.

Student 3 recognized that when groups were assigned, he was often the highest performing student, but he did not believe he had the ability to carry the weight of the group:

Student 3:… and then we were put in groups and I always had an A so when you started putting us in groups, I think with percentiles I was always the top quartile, but like I definitely did not know P. Chem. so I was always with three people who knew even less than me…

Most of these students felt differently when they were allowed to choose their own groups. Students 1 and 3 described clear strategies in which they would divide the work or problems based on the abilities in their groups. They expressed their comfort discussing the concept and their reasoning for their answers with their chosen peers. Student 2, however, said the dynamic between members when she was allowed to choose her group was “About the same. Because I know that this person can be charged to like, write it down, you know, that we can talk about it and there would still be someone who just lingers.” Student 5 had an opposing view. He discussed the fact that when students were allowed to choose their own groups “the people that knew what they were doing, they would usually congregate in their own group and everybody else is just shut out of that. So they’re not exactly sharing their wealth of knowledge with anybody else.”

Other students recalled group activities differently. Student 6 began by saying, “In class when we were presented with an activity, between group members you would see, um, pretty much others that were trying to help, um, the people in the group that didn’t understand the information.” She also expressed that working in assigned groups “wasn’t catastrophic, it was still a good experience to know that everybody…it's not just you in the class that didn’t understand it, there's other people that didn’t understand it or you know, there's other people that could show you how to do it in other ways…” This is a similar sentiment to several students on the survey as well as students in Pentecost's (2000) study. Student 4 described a similar strategy for group work to Students 1 and 3. However, she seemed to feel more like Student 6 in terms of the group dynamics:

Student 4: I mean, what I was trying to tell my friends when I was doing it, like each one do something and then we collect everything and, like, if you have a question, we can work on it, like, something hard – that's how I was trying to do it.

One important note to make is that students often felt rushed to complete the class activities. It was a common concern over the course of the semester, so the instructor made a practice to stay in the hallway outside of the classroom after the class time was up in order to give students time to finish writing down their answers. Students reacted differently to this time crunch. For example, Student 4 liked the challenge and thought it compelled her to problem solve:

Student 4: When we were doing the homework from the book and talking to each other in groups. It was, like, a little bit challenging – the no time and we were running and fighting, you know, one looking left, one looking right, you know what I mean? I like that. Because it makes you like, I don’t know, talk to your friends and then try to solve in groups. That's what I liked a lot.

However, Students 1, 3, and 5 felt that they were pressured to “put something on paper” and did not really learn the concept from the activity. In his discussion of working with assigned group members, Student 1 indicated that understanding and explaining the concepts was not his priority when there was a time crunch for finishing the assignments:

Student 1: …but then there were a couple times where I don’t think they, I don’t think my group mates would grasp the idea that we had five minutes to finish this, we need to get this done and they’d be like, “Wait. Can you explain it?”

Interviewer: And so did you explain it when somebody had a question?

Student 1: If we had time. If not, I told them to leave it for later because I needed to get the assignment done.

Student 3 tended to feel stressed:

Student 3: …and I didn’t know a lot to begin with so we’re all just like, “Alright we have 2 problems in 10 minutes and we have no idea what to do.” And it was always like, “All right let's put something on paper” so that was always pretty stressful too…”

Student 5 said that the lack of time would force them to find a superficial strategy to solve the problem and if a group member found an answer, they did not all necessarily understand and agree on the approach, rather, “It was all in on one answer.”

Other trends. Interestingly, a look at less frequent themes on the surveys reveals the effectiveness and enthusiasm of the instructor, reminding us that regardless of pedagogy, instructor beliefs and attitudes have a significant impact on students. In fact, during the interviews, students were also asked whether there was a particular concept they felt they understood better because of the teaching method. This was meant to probe corrected or lingering alternative conceptions, but interview results revealed that regardless of the concept the student mentioned, they felt they had a better understanding of it because of how it was explained by the instructor. Every student interviewed shared this view as shown below:

Student 1: Like when he would do it on the board in lecture, I would be like, “Oh my gosh. That just like…it clicked.”

–

Student 2: Well. I feel like I maybe got a better understanding of the equilibrium concept um…due to the way it was explained in class.

–

Student 3: I think dynamic equilibrium was one of them he explained that a lot better than I’ve learned it before and that made a lot of sense.

–

Interviewer: Ok and was there like a specific activity that we did that helped you understand that or was it just the way Dr. Hernandez was teaching?

Student 4: It was teaching. Like the energy one? Yeah, he was teaching it.

–

Interviewer: Was there anything that we did in the course that helped you really understand that?

Student 5: Explaining it in a way outside of mathematics.

–

Student 6: You know and the fact that he explained it in such detail…

Limitations

Phenomenological studies aim to reveal how students experience certain events. They do not necessarily reveal the “truth”. Thus, we have presented our best explanation of how these students experienced active learning in a Physical Chemistry I course covering thermodynamics and kinetics concepts. Students at different universities or with different professors may describe their experiences differently.

The observations were performed on only a group of ten students at a time so that a large enrollment course could be observed. However, it is important to remember that each observation does not necessarily represent what the students in every other group were doing during the class period. In addition, observers collected data on different groups of students, making it difficult to establish reliability using inter-rater reliability methods. However, these data merely corroborate the other evidence presented. They are not meant as standalone evidence of engagement.

The survey responses were coded using an inductive coding scheme which involves looking for themes without previously defining codes. However, both the first and corresponding author participated in the development and facilitation of the course, making coding impossible to do tabula rasa. In order to address any concerns about this coding procedure, we first express our belief that no research can truly be conducted tabula rasa. Second, we have presented the complete results of coding in Appendix 2 so that the reader can be assured that themes other than those corresponding to the course design were identified. We presented the chosen themes in this paper since they gave the most information about how students experienced the active learning strategies we applied.

Finally, students were asked via e-mail to volunteer to participate in interviews. The six students that volunteered all obtained A's in the course. Their perceptions may differ from students who did not receive an A or A- in the course. However, the diverse views expressed by these students can provide much insight into how students respond in an active learning environment.

Implications

The results presented herein can contribute greatly to the movement toward the implementation of active learning in upper-level chemistry courses. In particular, these results provide a useful resource for faculty, instructors, graduate teaching assistants, and other university educators when considering implementing active learning in the classroom. Obviously, students respond to these changes and their responses are important to keep in mind when developing an active learning course. Our students shared several characteristics with students from previous studies including increased feeling of responsibility for knowing concepts, manifestation of the “problem-solving mindset” which influenced their study strategies, and mixed emotions about group work (i.e. appreciation for different perspectives and resentment for lackluster group members). This provides validity to our findings and in tandem with our additional results (i.e. the combination of the strategies with an exam-free approach), provides implications for effectively implementing active learning in physical chemistry, improving this particular implementation and minimizing negative responses from students. These implications will be discussed in terms of each of the active learning strategies employed.

Alternative assessments. The dualistic response to alternative assessments suggests that a minor change in how we refer to assessments could both keep test anxiety low while maximizing motivation to perform. Rather than referring to out-of-class assignments as “homeworks”, we could refer to them as “take-home exams”. Assigning several take-home exams after the material has been covered in class may provide extrinsic motivation for the students to study, but reduce the anxiety associated with the lack of time and resources available on a traditional in-class exam.

Discussions. The students’ perception of discussions indicated that they felt accountable for understanding the information before class and that they learned from hearing their classmates’ perspectives. Survey and interview responses clearly show that students recognize the value in this exercise. In addition, random calling on students during the whole-group discussions was one strategy that allowed the instructor to ensure individual accountability in a course with no exams. Any negative responses to this strategy (i.e. anxiety about being called on), can be alleviated, but not likely eliminated, by clearly communicating the expectation that students participate in the discussion.

Homework/flipped classroom. There are a myriad of descriptions in the literature about flipping the classroom. The common underlying theme is that content delivery occurs outside of class and there is some sort of practice or application opportunity for students during class. However, the specific manifestation of each of these components is varied. For instance, as instructors, we wanted our students to read the material rather than to watch videos. The reason for this was two-fold. First, we prefer the presentation of the material in the manner it is presented in the textbook we chose. Second, we wanted to guide our students in learning by actively reading rather than watching videos in order to help them develop this skill. However, we do not believe that our results implicate one way or another that reading should be used as content delivery over videos. This was simply our preference and the same argument can be made for having in-class discussions followed by problem solving in groups during class. This aside, it has been established that this kind of instruction requires scaffolding. For this reason, the instructor modeled some active reading strategies on the first day of class and continued in-class discussions to help students focus on main concepts. Despite this, students expressed feeling overwhelmed by the reading.

Group work. A common student complaint about group work, and one which some of our students expressed, was that not all group members “pulled their own weight”. It is important that the instructor ensure individual accountability during group work to reduce these experiences and encourage learning among all group members. For this reason, we reassigned groups frequently (every week or every two weeks depending on how effectively we thought the groups were working together). Most often groups were assigned based on course performance. Seldom, students were allowed to choose their own groups. Assigning groups based on performance, but reassigning them frequently was intended to prevent students from relying on any single group member throughout the semester.

Conflicts of interest

There are no conflicts to declare.

Appendix 1: interview protocol

Appendix 2: coding categories

After organizing and becoming familiar with all of the survey responses, we freely developed several categories by making notes on the surveys to describe the content in as much detail as possible. We then grouped these categories under higher order headings, reducing the number of main categories as much as possible. The main category we report in the paper is “instructional strategies”. However, other main categories included “classroom environment”, “course focus” and “learning tools”. We chose to focus on the first category for two reasons. First, the sub-categories (alternative assessments, discussions, flipped classroom and group work) do align well with the structure of the course and, thus, would help us answer our research question. Second, the majority of responses fit in this category, so we chose to focus our analysis there. All coding categories are shown in the figure below.

Acknowledgements

We would like to acknowledge the UCF Initiatives in STEM iSTEM Fellows Program for financial support and the students of CHM3410 in Fall 2016 for participating.

References

1. Bain, K., Moon, A., Mack, M. R., & Towns, M. H. (2014). A review of research on the teaching and learning of thermodynamics at the university level. Chem. Educ. Res. Pract., 15(3), 320–335.
2. Becker, N., Stanford, C., Towns, M., & Cole, R. (2015). Translating across macroscopic, submicroscopic, and symbolic levels: the role of instructor facilitation in an inquiry-oriented physical chemistry class. Chem. Educ. Res. Pract., 16(4), 769–785.
3. Beichner, R. J., Saul, J. M., Abbott, D. S., Morse, J., Deardorff, D., Allain, R. J., Bonham, S. W., Dancy, M., & Risley, J. (2007). The student-centered activities for large enrolment undergraduate programs (SCALE-UP) project. Res.-Based Reform Univ. Phys.1(1), 2–39.
4. Bunce, D. M., Flens, E. A., & Neiles, K. Y. (2010). How long can students pay attention in class? A study of student attention decline using clickers. J. Chem. Educ., 87(12), 1438–1443.
5. Carter, C. S., & Brickhouse, N. W. (1989). What makes chemistry difficult? Alternate perceptions. J. Chem. Educ., 66(3), 223 DOI:10.1021/ed066p223.
6. Committee on Professional Training, (2015). Undergraduate Professional Education in Chemistry: ACS Guidelines and Evaluation Procedures for Bachelor's Degree Programs, Physical Chemistry Supplement. Retrieved from Washington, DC.
7. Credé, M., Roch, S. G., & Kieszczynka, U. M. (2010). Class Attendance in College. Rev. Educ. Res., 80(2), 272–295 DOI:10.3102/0034654310362998.
8. Creswell, J. W. and Poth, C. N., (2017), Qualitative inquiry and research design: Choosing among five approaches, 4th edn, Thousand Oaks: Sage.
9. Derrick, M. E., & Derrick, F. W. (2002). Predictors of Success in Physical Chemistry. J. Chem. Educ., 79(8), 1013 DOI:10.1021/ed079p1013.
10. Elo, S., & Kyngas, H. (2008). The qualitative content analysis process. J. Adv. Nurs., 62(1), 107–115 DOI:10.1111/j.1365-2648.2007.04569.x.
11. Fox, L. J., & Roehrig, G. H. (2015). Nationwide Survey of the Undergraduate Physical Chemistry Course. J. Chem. Educ., 92(9), 1456–1465 DOI:10.1021/acs.jchemed.5b00070.
12. Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci. U. S. A., 111(23), 8410–8415.
13. Gardner, D. E. and Bodner, G. M. (2007). Existence of a problem-solving mindset among students taking quantum mechanics and its implications, ACS Publications.
14. Hahn, K. E., & Polik, W. F. (2004). Factors Influencing Success in Physical Chemistry. J. Chem. Educ., 81(4), 567 DOI:10.1021/ed081p567.
15. Hinde, R. J., & Kovac, J. (2001). Student active learning methods in physical chemistry. J. Chem. Educ., 78(1), 93.
16. Lane, E. S., & Harris, S. E. (2015). A new tool for measuring student behavioral engagement in large university classes. J. Coll. Sci. Teach., 44(6), 83–91.
17. MasteryConnect (2018), Socrative, accessed 2 February 2018, https://www.socrative.com/.
18. Moustakas, C. (1994) Phenomenological research methods, Thousand Oaks: Sage.
19. Nicoll, G., & Francisco, J. S. (2001). An investigation of the factors influencing student performance in physical chemistry. J. Chem. Educ.(1), 99.
20. O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. Internet High. Educ., 25, 85–95 DOI:10.1016/j.iheduc.2015.02.002.
21. Partanen, L. (2016). Student oriented approaches in the teaching of thermodynamics at universities–developing an effective course structure. Chem. Educ. Res. Pract., 17(4), 766–787.
22. Pentecost, T. C., & James, M. L. (2000). Creating a student-centered physical chemistry class. J. Coll. Sci. Teach., 30(2), 122.
23. Seery, M. K. (2015). Flipped learning in higher education chemistry: emerging trends and potential directions. Chem. Educ. Res. Pract., 16(4), 758–768.
24. Sözbilir, M. (2004). What Makes Physical Chemistry Difficult? Perceptions of Turkish Chemistry Undergraduates and Lecturers. J. Chem. Educ., 81(4), 573 DOI:10.1021/ed081p573.
25. Strayer, J. F. (2012). How learning in an inverted classroom influences cooperation, innovation and task orientation. Learn. Environ. Res., 15(2), 171–193 DOI:10.1007/s10984-012-9108-4.
26. Towns, M. H., & Grant, E. R. (1997). “I believe I will go out of this class actually knowing something”: Cooperative learning activities in physical chemistry. J. Res. Sci. Teach., 34(8), 819–835.
27. Weaver, G., & Sturtevant, H. (2015) Design, Implementation, and Evaluation of a Flipped Format General Chemistry Course. J. Chem. Educ., 92(9), 1437–1448.
28. Xu, X., Villafane, S. M., & Lewis, J. E. (2013). College students’ attitudes toward chemistry, conceptual knowledge and achievement: structural equation model analysis. Chem. Educ. Res. Pract., 14(2), 188–200.

---