Identifying the challenging characteristics of systems thinking encountered by undergraduate students in chemistry problem-solving of gas laws

Abstract

Systems thinking has been an educational priority for more than a decade, yet its related assessment and teaching strategies have been understudied in the chemistry education research community. Through the lens of systems thinking, this study explores how undergraduate students connect and translate their conceptual representations when they are involved in contextualised problem-solving. The ‘Contextualised Problem Solving’ (CPS) assessment instrument contains four open-ended questions about gas law. Three different cohorts of students registered in a physical science course (2016 Fall, 2017 Spring, 2017 Fall semesters) participated in the problem-solving component of CPS. The results showed that only 8% of students were capable of higher order systems thinking ability when they engaged in problem solving. Over half of the students failed to retrieve essential concepts in problem situations. Most of the participants demonstrated difficulties in organising related systems’ components, understanding the cyclic nature of relationships among systems, and identifying limitations in a specific problem context. By identifying the difficulties and challenges of systems thinking experienced by undergraduate students in solving complex chemistry problems, these findings have the potential to provide fresh insights into effective teaching strategies to promote students’ higher order thinking skills.

---

Introduction

Systems thinking has come to the attention of chemistry educators more predominantly in the past few years (Hrin et al., 2017). It is viewed as a viable approach to convey not only the understanding of different components and processes of any system, but also how each of the components and processes affect other components and processes (Ben-Zvi Assaraf and Orion, 2010). Batzri et al. (2015) have defined systems thinking as the ability to deeply understand and interpret systems’ characteristics and behaviour. A systems thinker is capable of identifying systems’ components, recognising the relationships among them, exploring and understanding emergent properties of the components, and analysing and synthesising phenomena in a wider context (Ben-Zvi Assaraf and Orion, 2005). Although systems thinking studies have been widely conducted in biological systems (Wilensky and Reisman, 2006; Verhoeff et al., 2008); technological systems (Hong et al., 2013); and earth science systems (Ben-Zvi Assaraf and Orion, 2010), the applications of systems thinking in chemistry education is understudied (Hrin et al, 2017).

Requirements of success and students’ performance in solving complex chemistry problems has been documented in existing literature. For example, working memory capacity is important in relation to predicting students’ performance in science problem-solving. Working memory capacity represents the ability to activate relevant information and inhibit irrelevant information while engaging in problem solving (Solaz-Portoles and Sanjose-Lopez, 2009). According to Johnstone and El-Banna's predictive model in problem solving, a student is likely to be successful if the problem's mental demand is less than or equal to a student's working memory capacity (Johnstone, 1984; Johnstone and El-Banna, 1986). Additionally, many cognitive variables (e.g., mental capacity, field dependence and field independence, and the mobile/fixed cognitive style) are associated with students’ problem-solving performance in science (Solaz-Portoles and Sanjose-Lopez, 2009). Additional requirements for success in problem-solving proposed by the literature include: understanding the underlying concepts of the problem (Gabel and Bunce, 1994), engaging higher order cognitive skills (Zoller and Pushkin, 2007) and deploying metacognitive strategies (Cooper and Sandi-Urena, 2009). However, research studies on the assessment of student performance in problem solving reveal that a variety of difficulties are often encountered by students when they attempt to put this into practice. For example, in a study investigating prospective chemistry teachers’ mental models of vapour pressure, Tumay (2014) found that many participants experienced difficulty in identifying the relevant components, their properties, and their interactions in a liquid–vapour equilibrium system, and in tying these ideas in together accordingly. The difficulty experienced by the pre-service chemistry teachers in Tumay's study might be partially attributed to the higher-order thinking skills (HOTS) required in high-level systems thinking, such as the ability to visualize and synthesise system components (Ben-Zvi Assaraf and Orion, 2010). As these HOTS are often not explicitly taught in typical chemistry classrooms, it is perhaps unsurprising that undergraduate prospective chemistry teachers experience difficulties in applying these forms of skills for conceptualisation and problem-solving.

The chemistry topic of gas laws is often one of the more challenging units for students’ conceptual understanding and application. Consistent results across more than a decade of chemistry education research suggest that both high school and university students experience difficulties in the application of conceptual understanding and problem-solving of gas law test items (e.g., Lin et al., 2000; Coştu, 2007; Matijašević et al., 2016). Student difficulties and failures have been attributed to rote memorisation and misuse of chemical formula (Lin et al., 2000), or poor external representation and internal interpretation of a chemistry problem or phenomena (Matijašević et al., 2016). The difficulties may become more serious when they encounter a complex problem-solving situation which requires the ability to coordinate multiple representations, such as open-ended complex problems containing multiple variables (Overton et al., 2013). For example, a gas law problem may be composed of several connecting individual systems which can be open or closed systems and might contain liquids, solids, and a variety of gases in each system. Previous studies have identified that even pre-service chemistry teachers who already have substantial content knowledge and conceptual understanding still experienced difficulties in identifying the relevant components, properties, and interactions in a liquid–vapour equilibrium system, and in tying these ideas in together accordingly (Tumay, 2014). It is likely that additional barriers other than insufficient conceptual understanding are related to learners’ performance of problem solving. Consequently, further identification of exploring the challenging characteristics of systems thinking encountered by students exploring the topic of gas laws has the potential to provide fresh insight into effective teaching strategies to promote student chemistry problem-solving capability.

Theoretical framework and research questions

This study employed the eight hierarchical characteristics of systems thinking, described by Ben-Zvi Assaraf and Orion (2010), as a lens to examine the levels of university undergraduate students’ systems thinking capabilities. These are listed below:

(1) Identifying the components of a system;

(2) Identifying relationships among a system's components;

(3) Identifying dynamic relationships within the system;

(4) Organising the system's components and processes;

(5) Understand the cyclic nature of systems;

(6) Making generalizations;

(7) Understanding the hidden dimensions of the system; and

(8) Thinking temporally with retrospection and prediction.

The study was not designed to assess whether students’ answers to test items were correct or incorrect, but rather focused on identifying the hierarchical characteristics of students’ systems thinking in solving complex chemistry problems. The following research question guided this study: What are the difficulties and key gaps experienced by undergraduate students in transferring between contextualized problem-solving and individual conceptual representations through the lens of systems thinking?

Methods

Participants

This study took place at one comprehensive university in Kaohsiung city located in south Taiwan. The study was performed in compliance with the author institute's policy on research ethics. A total of 179 undergraduate students participated in this study. The students were in their sophomore, junior, and senior year of university study. The participants were from three different cohorts registered in a course named “The application of scientific inquiry in physical science” in 2016 Fall, 2017 Spring, and 2017 Fall semesters. The goal of the course was to foster students’ scientific literacy and apply science-related concepts to problem-solving in daily life. This 18 week course (two hours a week) was available for students across six colleges: Liberal Arts, Science, Engineering, Management, Marine Science, and Social Science. The three cohorts in 2016 Fall, 2017 Spring, and 2017 Fall semesters consisted of 64, 72, and 43 students respectively. These students were in the top 15% of performers in their college entrance examinations among the overall exam attendees.

Contextualised problem-solving assessment instrument (CPS)

The Contextualised Problem-Solving assessment instrument (CPS) used in this study contains four open-ended questions. It was used to understand how students make connections and translate between individual conceptual representations and problem situations. In solving the questions, students are not required to perform significant quantitative calculations or memorize formulas. In contrast, they have to construct and transfer their conceptual understanding and application of gas law in solving problems within different context-based situations. The four questions were reviewed by two science educators who were familiar with the topic of gas law and the contextualised assessment. They checked if each question met the criteria of examining the conceptual understanding and application of gas law (Charles’ law, Boyle's law, or atmospheric pressure). We also pilot tested to check the feasibility and the readability of the CPS with 72 undergraduate students who registered in the same course in the spring semester of 2016. The Cronbach's alpha reliability of the CPS was 0.77 in the pilot test.

Students responded to the instrument in the 9th week of the scientific inquiry course because the content and topic of the course in weeks 1 to 8 were related to the application of atmospheric pressure, liquid and gas pressure, and gas law. In order to make sure that students were equipped with essential content knowledge of gas laws, we provided demonstrations and hands-on activities to allow students the opportunity to consolidate the content knowledge that they had previously learned. For example, in reviewing Charles’ Law, the demonstration of a siphon coffee maker was used to show how the heating and cooling of the lower vessel affected vapour pressure. In reviewing Boyle's Law, a sealed balloon enclosed in a syringe was used for demonstration purposes (see Fig. 1 below). Students were invited to push or pull the piston and observe how the volume of the balloon is changed in line with pressure changes inside the cylinder. These are only brief examples of a wide range of activities that were provided to students to support their chemistry learning in contextualised and practical ways. After all content knowledge of gas laws had been reviewed, students were asked to answer the CPS. It took students 50–110 minutes to completely answer the four questions displayed in Fig. 2–5.

Fig. 1 Syringe and balloon tool to demonstrate Boyle's law.

Fig. 2 CPS Question 1.

Fig. 3 CPS Question 2.

Fig. 4 CPS Question 3.

Fig. 5 CPS Question 4.

Systems thinking scoring schemes of CPS

To examine students’ systems thinking levels of CPS, we modified Ben-Zvi Assaraf and Orion's (2010) original model because the complexities of our chemistry problems are different from the “water cycle on the earth system” of Ben-Zvi Assaraf and Orion's (2010) study. The following three hierarchical characteristics of scoring schemes are derived from Ben-Zvi Assaraf and Orion's (2010) original model:

(1) Characteristic 1: retrieving essential science concepts embedded in all components of a whole system:

With this characteristic, students should identify important components included in a given problem context. They should also make distinctions as to what subsystems (e.g., internal and external systems) are included in a whole system.

(2) Characteristic 2: organising the relationships among scientific concepts and subsystems.

With this characteristic, students should identify the intraspecific relationship in each subsystem (e.g., the height of water in a container has its own pressure inside the system, or there is external atmospheric pressure outside of the container). In addition, students should be able to indicate dynamic or interdependent relationships among the science concepts within each subsystem (e.g., PV = constant in a closed subsystem).

(3) Characteristic 3: identifying hidden dimensions and limitations of a whole system and making generalisations.

While making generalisations, students should be able to recognise patterns or interrelationships among subsystems which are not necessarily readily apparent, and be aware of the requirements and restrictions in a given problem context (e.g., closed system, fixed volume etc.)

Overall, in solving complex chemistry problems, students must analyse and integrate the whole system (problem situation) to be successful in problem-solving. Firstly, they need to identify the key components and related science concepts and all subsystems of the whole system. Then, they must organise the relationships or links between these components and subsystems. Finally, they have to recognize limitations and make generalizations with consideration for the whole system.

Full statements for how these characteristics manifest in each of the four questions in Fig. 1–4 are described as follows.

For Q1, in order to explain why the water can enter the body of the pottery cow or the water inside the pottery cow spray out, students should apply the concept of atmospheric pressure and identify that when the volume of a gas is held constant, its temperature and the pressure will be in direct proportion. Firstly, students should point out that the volume and quantity of gas are held constant in a hollow and closed pottery cow. Secondly, they should explain that the gas pressure inside the cow's hollow body would decrease when the cow is immersed into a tank of cold water as this becomes a closed system and the temperature decreases. At this point, the atmospheric pressure is greater than the cow's internal gas pressure and limits the ability to push out the water entering into the body of the cow. When hot water is then showered over the top of the cow, this results in a rise in temperature and an increase in the gas pressure of the almost closed system. Consequently, the high internal gas pressure pushes the water inside the cow's body causing it to spray out since the internal pressure is now much greater than the atmospheric pressure.

In Q2, students are expected to retrieve the concepts of atmospheric pressure and water pressure, and be able to identify the differences in the depth of water level to predict whether water in the three bottles would flow out or not. Firstly, it is important to identify that the waters in Bottle I, II, and III are initially in closed, closed, and open systems respectively. Secondly, organising and comparing the relationship between internal and external pressure for each small hole in the three bottles are critical. In Bottle I, the external atmospheric pressure of A and A′ holes are the same. While the water depth levels for point A and A′ are the same, the water pressure of A and A′ should be similar. In addition, given that the water level is only about 20 centimetres (which is far less than the 10.3 metres height that the standard atmospheric pressure can hold), the water would not flow out from hole A or A′. In Bottle II, the external system of the bottle's atmospheric pressure exerting on holes A and B are the same. However, the internal system's water pressure of hole A is bigger than hole B because of the difference of the two holes’ water depth. In other words, the total internal system pressure of point A proceeding towards outside equals to the sum of about 15 centimetres height (i.e., the distance between A and B) of water pressure and the atmospheric pressure exerted on hole B, while the external system pressure proceeding toward inside the bottle has only the standard atmospheric pressure. Consequently, water would flow out from hole A. In Bottle III, air enters the bottle from hole C. The bottles’ internal system has a standard atmospheric pressure along with the water pressure proceeding towards the outside of hole A. As the bottles’ external system pressure contains only the atmospheric pressure proceeding towards the inside of the bottle, the water would flow out from hole A.

In Q3, the change of gas pressure is the major reason that causes water to flow out. First, the water level in both containers A and B and the lengths of no. 1–4 glass tubes should be drawn correctly. In addition, explanations should point out that container A will become a closed system when water is poured into the funnel from above, and container A and B are connected with no. 2 and 3 glass tubes. Secondly, acceptable explanations should indicate that the gas volume of container A will decrease because the water level of container A rises. Consequently, the gas pressure of container A increases according to Boyle's law (PV = constant) and further pushes the water into container B, causing it to flow out.

In Q4, students are expected to apply the following concepts: atmospheric pressure, water pressure, and the changes of gas pressure and volume. In item (a), the first reason why water doesn’t flow out from the small holes of the gauze is the surface tension maintained while the water level is kept horizontal. The second reason is that the water pressure inside the plastic cup is smaller than the external atmospheric pressure outside of the plastic cup. In item (b), it is a closed system inside the plastic cup. The gas volume enclosed in the plastic cup has increased when water was squeezed out. Therefore, the gas pressure inside the plastic cup would decrease according to Boyle's law (PV = constant). It gradually achieves equilibrium between the internal total pressure of the gas pressure and the water pressure inside of the plastic cup, and the external atmospheric pressure outside of the plastic cup (i.e., external atmospheric pressure = internal water pressure + gas pressure). Therefore, water in the plastic cup would not flow out eventually.

Students’ answers to these questions were coded individually by two well trained science educators according to the above explanatory statements. Coding scores for each of the three hierarchical characteristics (retrieving science concepts, organising concepts systematically, and identifying hidden dimensions and making generalisations) were classified from zero to three points according to the completeness of each answer. The full possible score for each question was 9 points. The total score of CPS is 36 points. The inter-rater reliability of CPS on the four questions ranged from 0.937 to 0.959, p < 0.01. Student answer examples for each characteristic scores are provided in Table 1.

Table 1 Examples for each characteristic scores

Characteristics	Score	Description	Example^a
a In this table, Q1–4 refers to CPS Questions 1–4. In Q1, students were asked to explain why the water would enter or spray out of the body of a pottery cow when it was immersed into cold water, or showered with hot water. In Q2, students were requested to describe if the water would flow out from the drilled holes in three plastic bottles. In Q3, students were asked to draw the water level in containers A and B and the lengths of glass tubes, and explain what process causes the water to flow out continuously. In Q4, students needed to explain why the water would not flow out of a plastic cup which was covered with gauze. The context-based situations of each question are described in detail in Fig. 2–5.
Retrieving essential science concepts embedded in all components of a whole system	0	Incorrect, unrelated or no retrieval	(Q2 2017Spring Student40) The water would not flow out because the pressure on both A and A′ holes are the same. (Lack of explanation about what pressures are the same.)
	1	Minor retrieval	(Q4 2016Fall Student11) The atmospheric pressure pressed on each hole of the gauze. (Only mentions atmospheric pressure.)
	2	Major retrieval	(Q2 2017Spring Stuent57) The water would not flow out of Bottle I. The water pressure at both A and A’ holes are the same due to the same depth of the water. (Lack of atmospheric pressure concept.)
	3	Full retrieval	(Q1 2017Fall Student34) As the temperature dropped, the volume of gases inside the cow became smaller, causing the decrease of its gas pressure. The water would flow into the pottery cow because the pressure of gases is less than the atmospheric pressure.
Organising the relationships among scientific concepts and subsystems	0	Incorrect, unrelated or no organisation	(Q3 2017Fall Student06) When pouring water into container A, the pressure in container A is equal to the total of gas pressure plus water pressure. There is only gas pressure in container B. The pressure difference between the two containers makes the water flow out. (Wrong explanation of pressure difference.)
	1	Minor organisation	(Q4 2016Fall Student60) The water would not flow out because the water pressure is less than the pressure of gases. Squeezing the cup caused the water flowing out and air entered into the cup. The water would not flow out because of new pressure equilibrium. (Lack of explanation about new equilibrium.)
	2	Major organisation	(Q1 2016Fall Student42) After the pottery cow was put into hot water and then placed into cold water, the water would enter the pottery cow because the pressure of gases inside the pottery cow is less than atmospheric pressure. Then, pouring hot water on the pottery cow, the water would spray out because the pressure of gas inside the pottery cow is bigger than atmospheric pressure. (Lack of description about the relationship between gas pressure changes and temperature changes inside the pottery cow.)
	3	Full organisation	(Q3 2017Spring Student18) When we poured water into container A, the volume of gases decreased because of the water level rising. According to Boyle's law, the pressure of gases would increase because its volume decreased and pushed the water flow out from tube 4 as container A and B are in a connected closed system.
Identifying hidden dimensions and limitations of a whole system and making generalisations	0	No identification	(Q4 2017Spring Student15) The water would not flow out. The water pressure is less than atmospheric pressure because it could support a 10 metre height of water. If we did not apply force on the cup, the water would stop flowing out. (The student did not identify any hidden dimensions or limitations, like closed system, surface tension, etc.)
	1	Minor identification	(Q3 2016Fall Student58) The volume of gases decreased as the volume of water increased. Since container A was a closed system, the gas pressure would become bigger due to its volume decrease (PV = K) and then push the water into container B, causing it to flow out. (The student merely identified a closed system and made an incorrect generalisation about the water level in container A and B, and the lengths of tube 1–4.)
	2	Major identification	(Q4 2016Fall Student35) Apart from surface tension, the main reason is that atmospheric pressure is bigger than the water pressure inside the cup as the atmospheric pressure could support 1033 cm height of water. While squeezing the cup, the water flows out and air enters into the cup. The pressure of gas decreased because the gases volume inside the closed cup increased (Boyle's law). The water would not flow out because atmospheric pressure outside the cup is bigger than the pressure of gas inside the cup. (The student did not mention the limitation relating to the water level being kept horizontal.)
	3	Full identification	(Q1 2017Fall Student17) When the pottery cow was put into cold water, it became a closed system. The volume and quantity of gases inside the pottery are fixed. The pressure of gases inside the pottery would decrease because the temperature dropped. The water would flow into the pottery cow because the pressure of gas is less than atmospheric pressure outside of the pottery cow according to PV = nRT. When hot water was poured on the pottery cow, for the same reason, the pressure of gas inside the pottery would increase because the temperature rose. The water would spray out because the pressure of gas is bigger than atmospheric pressure. It is similar to the principle of a vacuum coffee maker.

---

Data analysis

In order to better understand students’ challenges and difficulties in solving contextualised problems, students were divided into three levels of systems thinking (i.e., low, medium, and high) in line with their scores on CPS. We also counted the percentage distributions of students’ performances for each level on the three hierarchical characteristics of systems thinking.

For qualitative data analysis, we identified the most common challenges and difficulties faced by students and provided exemplar student answers to explain how students erroneously used unrelated science concepts, failed to identify all interactive subsystems involved in the problem situation, struggled in clarifying the relationship among subsystems, and demonstrated difficulties in considering hidden dimensions and limitations corresponding to particular problem situations while making generalisations.

Results

Research Question: What are the difficulties and key gaps experienced by undergraduate students in transforming between contextualised problem-solving and individual conceptual representations through the lens of systems thinking?

One of the purposes of this study was to examine undergraduate students’ challenges in solving contextualised chemistry problems through the lens of systems thinking. Table 2 presents the percentage distributions of students’ performance at different levels of systems thinking. It can be seen that only 8% of the overall students’ systems thinking performance was able to be categorized as high level. These results indicate that students’ performance still needs to be improved when they apply their conceptual representations to contextualised chemistry problem-solving. We further analysed students’ performance on the three characteristics of systems thinking (i.e., retrieving science concepts, organising concepts systematically, and identifying hidden dimensions and making generalisations). The results are shown in Table 3 and demonstrate that students perform best on the characteristic of retrieving essential concepts embedded in all components of the whole system. A total of 49% of the participants performed well for this characteristic at a high level, which is considerably better than their performance in organising the relationships among the science concepts and subsystems (only 9% of participants performed at high level), and the performance of identifying hidden dimensions and limitations of a whole system and making generalisations (2% at high level). After analysing students’ responses quantitatively, we engaged in a more qualitative exploration of the challenges encountered by undergraduate students when they engage in the higher order systems thinking required to successfully respond to contextualised problem-solving activities. The most commonly found challenges for each characteristic in contextualised problem-solving are presented in Table 4 (only representing percentages over 15%).

Table 2 The numbers and percentages of students at different levels of systems thinking

Level of systems thinking	Numbers (%)
	2016 Fall (n = 64)	2017 Spring (n = 72)	2017 Fall (n = 43)	Total (n = 179)
a The numbers in parentheses are the range of scores of students’ performance in systems thinking. The highest possible total is 36.
Low (0–12)^a	18 (28%)	15 (21%)	12 (28%)	45 (25%)
Medium (13–24)	43 (67%)	52 (72%)	24 (56%)	119 (67%)
High (25–36)	3 (5%)	5 (7%)	7 (16%)	15 (8%)

---

Table 3 The percentage of students scored at low, medium and high levels for the three characteristics of systems thinking on CPS

Characteristic	Level	Numbers (%)
		2016 Fall	2017 Spring	2017 Fall	Total
Retrieving essential science concepts embedded in all components of the whole system	Low (0–4)	3 (5%)	5 (7%)	3 (7%)	11 (6%)
	Medium (5–8)	28 (44%)	30 (42%)	22 (51%)	80 (45%)
	High (9–12)	33 (52%)	37 (51%)	18 (42%)	88 (49%)
Organising the relationships among science concepts and subsystems	Low (0–4)	43 (67%)	38 (53%)	23 (53%)	104 (58%)
	Medium (5–8)	18 (28%)	27 (38%)	13 (30%)	58 (33%)
	High (9–12)	3 (5%)	7 (10%)	7 (16%)	17 (9%)
Identifying hidden dimensions and limitations of a whole system and making generalisations	Low (0–4)	41 (64%)	43 (60%)	24 (56%)	108 (60%)
	Medium (5–8)	21 (33%)	29 (40%)	18 (42%)	68 (38%)
	High (9–12)	2 (3%)	0 (0%)	1 (2%)	3 (2%)

---

Table 4 The most commonly found challenges for each characteristic in CPS

Characteristic	Challenge	Description
Retrieving essential science concepts embedded in all components of the whole system	Intuitively apply wrong or unrelated concepts	Q1: 16% of students believed that gas molecules would expand or contract when the temperature increased or decreased in a fixed-volume system
		Q3: 20% of students considered the change of gas volume and water pressures (or atmospheric pressure) as the main reason to make water flow out, rather than gas pressure in container A.
	Only identify partial concepts or subsystems in a whole system.	Q1: 34% of students focused on gas pressure inside the pottery cow, but neglected the presence of atmospheric pressure outside of the pottery cow.
		Q2: 41% of students identified atmospheric pressure, but paid no attention to the difference between the depth of the two holes’ water level.
		Q3: 32% of students recognised the change of gas pressure, but did not notice the change of gas volume.
		Q4: 45% of students focused on atmospheric pressure outside of the plastic cup, but neglected the gas pressure inside the plastic cup.
Organising the relationships among science concepts and subsystems	Fail to accurately express about the dynamic or interdependent relationships amongst all science concepts within a system	Q1: 61% of students failed to explain the interdependent relationship between the change of gas pressure and temperature inside the pottery cow.
		Q3: 58% of students did not point out the inverse relationship between gas pressure and gas volume in a closed system.
		Q4: 73% of students failed to express the inverse relationship between the pressure and the volume of the gas.
	Merely explain the dynamic relationships among components within an individual subsystem, but failed to clarify the specific relationships among different subsystems.	Q1: 25% of students understood the dynamic relationships between gas pressure and temperature, but failed to compare the internal system's gas pressure with the external atmospheric pressure.
		Q2: 44% of students could point out the interdependent relationships between water pressure and the depth of water level, but made mistakes when comparing internal water pressure with the external atmospheric pressure.
	The lack of competency in identifying the complexity of systems and clarifying the relationships among subsystems.	Q2: 53% of students were able to clearly illustrate the relationship between atmospheric pressure and water pressure in Bottle III. However, only just 16% of students made correct predictions and explanations about Bottles I and II.
		Q4: 60% of students were able to correctly describe the relationship between water pressure and atmospheric pressure. However, only 20% of students could properly explain the relationships among atmospheric pressure outside of the cup, water pressure, and the gas pressure inside the cup.
Identifying hidden dimensions and limitations of a whole system and making generalisations	Struggle to consider hidden dimensions and limitations.	Q1: 83% of students didn’t perceive that the volume of the gas inside the pottery cow is fixed.
		Q2: 85% of students could not recognize the closed system of Bottle II
		Q3: 68% of students could not point out how a closed system has been formed.
		Q4: 74% of students didn’t identify the contribution of surface tension in holding the water securely in the cup.

---

Characteristic 1 – retrieving essential science concepts embedded in all components of a whole system

For the characteristic of retrieving science concepts, over half of the students in this study (51%) failed to retrieve related science concepts in problem situations. The results imply that these students are not able to appropriately transfer their conceptual representations for problem-solving, even though they have learned about gas laws. The most common challenge encountered by students in retrieving science concepts is that students intuitively apply wrong or unrelated concepts to explain phenomena described in the problem situation. For example, in Question 1 (Q1), 20% of students wrongly use unrelated concepts (such as the small hole of the pottery cow would become bigger, or the volume of water inside the pottery cow expanded because the temperature increased) to explain why the water entered or sprayed out of the body of the pottery cow, as indicated in the sample responses below.

(Q1 S2017Fall Student14) At the beginning, the size of the small hole on the mouth of the pottery cow would become bigger because the temperature increased when it was put into hot water. At this time, when the pottery cow was placed in cold water, water would easily flow into the pottery cow for the reason that the space inside the cow and the small hole have become bigger. (This student wrongly identified unrelated concepts and components of the whole system for problem-solving. In actuality, the flow of water is caused by the invisible gas pressure or atmospheric pressure, rather than the size of the visible hole on the pottery cow).

(Q1 S2016Fall Student04) The volume of the cold water within the pottery cow would be expanded since the temperature rose when hot water was poured on the cow. As the space inside the pottery cow is fixed, the expanded water would be forced out of the cow.

Among those students who failed to appropriately retrieve related science concepts, 16% of them believed that gas molecules would expand or contract in volume when the temperature increases or decreases in a volume-fixed system, as indicated in the student sample response below:

(Q1 S2017Spring Student55) The volume of gas within the pottery cow expanded because the temperature increased when we immersed it in hot water, and then the gas would be contracted because the temperature decreased when it was put into cold water. The water would enter into the pottery cow to replace and fill the space that was previously occupied by the expanded gas. In the same way, the water would be pushed out when the pottery cow was covered by hot water. This is due to the space within the pottery cow not being enough to contain expanded gas.

In Q3, approximately 20% of students describe the major reason that water flows out from container B as being due to the change of gas volume and water pressures (or atmospheric pressure), rather than gas pressure in container A, as seen in student responses below:

(Q3 S2017Fall Student17) The gas inside container A and B belong to the same closed system. The water level of container A would rise after pouring into water from No. 1 glass tube. The gases in container A would run into container B resulting in the volume of gases in container B increasing. Then water would flow out to provide space to accommodate the additional gases.

(Q3 S2016Fall Student29) Adding water made the level of water in container A rise. Because container A and B were closed systems that connected to each other, the pressure of water column in No. 1 glass tubes pushed “water” from container A to B resulting in the height of the water rising in container B. Then, the increased pressure of the water forced water to flow out from the No. 4 glass tube.

Another challenge encountered by students in characteristic 1 of systems thinking was that students were only able to identify partial concepts or subsystems in a whole system. Students tended to simply notice a part of required concepts in a particular situation, and ignored the existence of other important concepts or subsystems. For example, in Q1, 34% of students merely focused on the changes inside the pottery cow (internal system) and were able to explain this appropriately, yet they neglected the presence of atmospheric pressure outside the pottery cow (external system):

(Q1 S2017Spring Student52) The air inside the pottery cow was hot gases because it was heated originally in hot water. When it was put into cold water, the pressure of gases decreased due to the temperature of hot gases dropping rapidly. So, water would enter the interior of the pottery cow. When the hot water was poured on the pottery cow, the pressure of gases increased due to the temperature of gases rising resulting in water being pressed out of the hole.

In Q4, nearly 45% of students recognised the presence of atmospheric pressure outside the plastic cup (external system), but paid no attention to gases pressure inside the plastic cup (internal system):

(Q4 S2017Spring Student12) In addition to surface tension, the water stopped flowing out because the water pressure inside the cup is less than atmospheric pressure.

Characteristic 2 – organising the relationships among science concepts and subsystems

Even if students demonstrated the ability to retrieve correct concepts and apply them to specific situations to solve a problem, they were often unable to clarify the relationships amongst the science concepts and subsystems. We found that although the percentage of students achieving high levels on retrieving essential science concepts was 49%, the percentage of students with high level abilities in organising the relationships among components and subsystems was only 9%.

The major challenge encountered by students in characteristic 2 appeared to be that they failed to accurately articulate and express their understanding about the dynamic or interdependent relationships amongst all science concepts within a system. For example, in Q1, only 39% of students could correctly explain the interdependent relationship between gas pressure change inside the pottery cow and temperature change. A common misunderstanding affecting 33% of students was the idea that changes in gas pressure in the pottery cow would have an interdependent relationship with changes in gas volume, as evidenced below:

(Q1 S2017Spring Student70) Volume changes would cause changes in gas pressure. The gas pressure within the pottery cow reduced to less than atmospheric pressure outside the pottery cow due to the fact that the volume of gases became smaller. Therefore, atmospheric pressure would push water into the pottery cow.

(Q1 S2017Fall Student24) The pottery cow was a closed space. According to PV = constant, the gas pressure decreased when the pottery cow was heated because the gas volume expanded. The gas pressure inside the pottery cow in hot water was less than in cold water, so water would run into the pottery cow. If we poured hot water on the pottery cow again, the gas volume inside the pottery cow would expand once more, thus water would be pushed out of the pottery cow to reduce the gas pressure.

In Q3, more than half (58%) of students did not point out the inverse relationship between gas pressure and gas volume in a closed system. 18% of students wrongly interpreted that the quantity and weight of gas forced the water to flow out continuously from container B:

(Q3 S2017Fall Student02) After adding water from the funnel, the water level in container A rose. The area of air above the container A and B and No. 2, 3 glass tubes formed a closed system. When the water level rose, the air in container A went through No. 2 and 3 glass tubes to container B. The quantities and weight of gases above container B increased. Consequently, the rise of gas pressure pressed on the water, so the water was forced out from the No. 4 glass tube.

A second challenge faced by students in characteristic 2 of systems thinking was that they merely explained the dynamic relationships among components within an individual subsystem, but failed to clarify (or at times even misinterpreted) the specific relationships among different subsystems. For example, in Q1, students demonstrated the ability to describe the dynamic relationships between gas pressure and temperature, but failed to compare the internal system's gas pressure with the external atmospheric pressure:

(Q1 S2017Fall Student25) At the beginning, both pressures in the pottery cow and on the outside of it were 1atm. When the pottery cow was put into cold water after it was heated in hot water, the gas pressure within the pottery cow decreased due to the drop of temperature. At this time, water flowed into the pottery cow because the gas pressure in the pottery cow was less than the surrounding water pressure.

In Q2, while students were able to successfully explain the interdependent relationships between water pressure and the depth of water level for the two different holes on each of the three plastic bottles, they showed a tendency to make mistakes about the relationships between water pressure inside the bottle (internal system) and the atmospheric pressure existing outside of the bottle (external system):

(Q2 S2017Spring 26) In Bottle I, water would flow out from the holes of A and A′. The water pressures at the point of A and A' holes are equal due to the fact that they have the same depth of water. However, Bottle I was full of water and the water pressure is bigger than atmospheric pressure, so water would flow out from both A and A' holes. In Bottle II, water would flow out from A and B holes. The water pressure at point A is bigger than at B because of their differences in the depth of water level (A was below B). Bottle II was filled with full water and the atmospheric pressure was less than the water pressure of A and B. Thus, water would flow out from both A and B holes and water from A hole would spray farther than from B hole. In bottle III, water would not flow out since C hole is open and the air inside and outside of the Bottle III could circulate with each other. The atmospheric pressures inside and outside of the bottle were at equilibrium state, so no water flowed out from either of the two holes.

The third challenge encountered by students in characteristic 2 of systems thinking was the lack of competency in identifying the complexity of systems and clarifying the relationships among subsystems. For example, in Q4, 60% of students were able to correctly describe the relationship between water pressure inside the cup which is filled with water, and the atmospheric pressure outside of the cup. However, students failed to perceive changes in gas pressure and gas volume above the water level when they were further asked to answer why water would not flow out continuously. Thus, only 20% of students were seen to be able to properly explain the relationships among atmospheric pressure outside of the cup, water pressure, and the gas pressure inside the cup. In Q2, more than half (53%) of students were able to clearly illustrate the relationship between atmospheric pressure and water pressure in Bottle III. However, students struggled to deduce the result for Bottles I and II, with only just 16% of students making correct predictions and explanations about these results.

Characteristic 3 – identifying hidden dimensions and limitations of a whole system and making generalisations

In relation to the third characteristic, our study indicated that most students struggled to consider hidden dimensions and limitations corresponding to problem situations in order to make informed generalisations. We found only 2% of students could perceive the requirements and restrictions of the whole system at a high level. For example, in Q3, only 32% of students could point out that the requirement to make water flow out continuously in a closed system must be formed. In Q4, the main reasons for water not flowing out from small holes of the gauze are (1) surface tension, and (2) the plastic cup being kept horizontal. There were only 26% and 35% of students who noticed these two limitations, respectively.

In summary, the results suggest that students have difficulties in organizing the relationships among the related science concepts in subsystems, and in identifying hidden dimension and limitations of a whole system. Overall, the lack of these higher order thinking skills is likely to cause inaccurate interpretations and subsequent erroneous generalizations from problem-solving.

Discussion

In the context of undergraduate chemistry problem-solving, the results of this study showed that only 8% of students were able to demonstrate high level systems thinking skills. Students performed quite well at retrieving essential science concepts embedded in a problem, but struggled with organising relationships among scientific concepts and subsystems, and did very poorly in identifying hidden dimensions and making generalisations. These concerning results might be attributed to a lack of prior experience in developing systems thinking skills. Avargil et al. (2012) claim that the teaching and development of higher-order thinking skills is rarely present in traditional chemistry instruction. Traditional teaching and assessment practices requiring algorithmic thinking and rote memorization of concepts, facts, or chemical formulas do not support the development of systems thinking skills (Fahmy and Lagowski, 2014). The findings of this study sit outside of our expectations of the capabilities of undergraduate students, and might act as a reminder to chemistry educators that simply focusing on the teaching of subject content knowledge may not be the best way of fostering competent chemistry learners. As Chang and Tzeng (2018) have concluded, “content knowledge plays only a conditional role in students developing visualization competence of that content. Beyond a certain point, merely increasing knowledge in a subject area may not result in higher visualization competence of that subject area” (p. 1223). In addition to assisting students to develop content knowledge, facilitating opportunities to practice higher order thinking skills such as systems thinking are critical for enhancing students’ problem-solving capabilities, both in their professional and personal lives.

This study's identification of difficulties and challenges encountered by undergraduate students in systems thinking provides some preliminary insight into the critical need to include more holistic and high level thinking and learning opportunities for students in chemistry education. While science education curriculum and research has begun to direct more focus towards the importance of promoting students’ systems thinking skills (e.g., Jacobson and Wilensky, 2006; Ben-Zvi Assaraf and Orion, 2010; NGSS Lead States, 2013; Tekkumru-Kisa et al., 2017), the results of this study reveal that even capable undergraduate students, who are front runners in terms of academic achievement, are still not well equipped to operate at this high level of competency. While the cross-sectional design of similar assessments for three different cohorts of students, along with the consistent findings, allows us to better understand the challenges and difficulties faced by students in the assessment of complex open ended problem-solving, caution must be taken not to over-generalise based on the findings. Among the limitations of this study are those related to its methodology and the participants sampled. Firstly, the students assessed in this study are a small group who self-registered in this course and chose to participate in the study. They are not fully representative of the whole population of capable university students. Secondly, while the results can present a meaningful first step toward the more complex context of systems thinking assessment, the results should not be over generalized to other units, topics, or science subjects.

In addition, research in this area makes apparent that there are many factors that may influence students’ ability to effectively engage in problem-solving processes. For example, students’ cognitive developmental stage (as understood within Piagetian theory), working-memory capacity, mental capacity, and field dependence/independence all contribute to student performance in problem-solving (e.g., Johnstone and El-Banna, 1989; Tsaparlis, 2005; Overton and Potter, 2011; St Clair-Thompson et al., 2012). In the context of this study, it was assumed that due to participants being drawn from the top 15% of performers in their college entrance examination, they would perform well in terms of higher order thinking skills and abstract reasoning, however the results of this study indicate that their cognitive capacities may vary significantly.

Another area of interest relates to the types of problems provided to students as this is also known to affect problem-solving performance. Studies indicate that students’ performance is often lower on solving conceptual questions requiring higher order cognitive skills (HOCS), and higher in solving algorithmic questions requiring lower order cognitive skills (LOCS) (Salta and Tzougraki, 2012; Broman et al., 2015). Given that the purpose of this study was to understand students’ level of systems thinking when engaged in complex problem-solving, we did not investigate how students perform on tasks requiring lower order cognitive skills, but this could potentially provide an interesting comparison.

Along with considering the role of cognitive factors, there is evidence that affective factors such as attitude towards the subject, students’ motivation, interest, emotion, and self-efficacy also play a significant role in learning chemistry (e.g., Taasoobshirazi and Glynn, 2009; Löffler et al., 2018). This study did not measure student affect towards the study of chemistry, however, exploring if there is any relationship between positive/negative affect and performance in problem solving tasks could be of interest for further research.

Implications for chemistry teaching and assessment

Despite the limitations of the study, the quantitative and qualitative findings work to demonstrate that even capable undergraduate students might still be under-developed in terms of systems thinking skills. The small percentage (<10%) of undergraduate students participating in this study being scored at a high level in characteristics 2 and 3 of systems thinking, (namely organising the relationships among science concepts and identifying hidden dimensions and making generalizations), deserves attention from chemistry educators. In addition to the traditional teaching of content knowledge, we may need to pay greater attention to how we can facilitate growth in students’ higher level systems thinking skills. The possible nature of effective teaching interventions in this area will be the target of our future research. In relation to assessment, incorporation of the design of contexualised and open ended test items based on a systems thinking framework, and discussion of the results of student engagement with these items, contributes to the currently small knowledge base about systems thinking assessment in chemistry education. These test items might also be further used as classroom discussion materials for students to reflect on their own responses, and gain appropriate feedback and advice.

Conclusion

This study has aimed to contribute to a better understanding of students’ difficulties in solving complex contextualised problems, which will hopefully make chemistry educators more aware of the shortfalls in students’ higher order thinking skills in the context of gas law problems. Our identification of different patterns of student challenges and difficulties in systems thinking might also lead to the design of teaching practices that provide focused learning opportunities for students to (1) identify and organize the interactive relationships among science concepts embedded in all related sub-systems, and (2) recognize hidden dimensions and limitations corresponding to problem situations while making informed generalisations.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

This work was supported by the Taiwan Ministry of Science and Technology (MOST) under Grant number MOST 107-2511-H-110-001.

References

1. Avargil S., Herscovitz O. and Dori Y. J., (2012), Teaching thinking skills in context-based learning: teachers’ challenges and assessment knowledge, J. Sci. Educ. Technol., 21(2), 207–225.
2. Batzri O., Ben-Zvi Assaraf O., Cohen C. and Orion N., (2015), Understanding the earth systems: expressions of dynamic and cyclic thinking among university studies, J. Sci. Educ. Technol., 24(6), 761–775.
3. Ben-Zvi Assaraf O. and Orion A., (2005), Development of system thinking skills in the context of earth system education, J. Res. Sci. Teach., 42(5), 518–560.
4. Ben-Zvi Assaraf O. and Orion N., (2010), Four case studies, six years later: developing system thinking skills in junior high school and sustaining them over time, J. Res. Sci. Teach., 47(10), 1253–1280.
5. Broman K., Bernholt S. and Parchmann I., (2015), Analysing task design and students' responses to context-based problems through different analytical frameworks, Res. Sci. Technol. Educ., 33(2), 143–161.
6. Chang H. Y. and Tzeng S. F., (2018), Investigating Taiwanese students’ visualization competence of matter at the particulate level, Int. J. Sci. Math. Edu, 16(7), 1207–1266.
7. Cooper M. M. and Sandi-Urena S., (2009), Design and validation of an instrument to assess metacognitive skillfulness in chemistry problem solving, J. Chem. Educ., 86(2), 240–245.
8. Coştu B., (2007), Comparison of students’ performance on algorithmic, conceptual and graphical chemistry gas problems, J. Sci. Educ. Technol., 16, 379–386.
9. Fahmy A. F. M. and Lagowski J. J., (2014), Systemic assessment as a new tool for assessing students learning in chemistry using SATL methods: systemic matching [SMQS], systemic synthesis [SSYNQS], systemic analysis [SAQS], systemic-analytic [SSYN-ANQS], as systemic questions type, Afric. J. Chem. Educ., 4(4), 35–55.
10. Gabel D. L. and Bunce D. M., (1994), Research on problem solving: chemistry, in Gabel D. L. (ed.), Handbook of Research on Science Teaching and Learning, New York: Macmillan, pp. 301–326.
11. Hong J. C., Chen M. Y. and Hwang M. Y., (2013), Vitalizing creative learning in science and technology through an extracurricular club: a perspective based on activity theory, Think. Skill. Creat., 8, 45–55.
12. Hrin T. N., Milenkovic D. D., Segedinac M. D. and Horvat S., (2017), System thinking in chemistry classrooms: the influence of systematic synthesis questions on its development and assessment, Think. skill. Creat., 23, 175–187.
13. Jacobson M. J. and Wilensky U., (2006), Complex systems in education: Scientific and educational importance and implications for the learning sciences, J. Learn. Sci., 15, 11–34.
14. Johnstone A. H., (1984), New stars for the teacher to steer by? J. Chem. Educ., 61(10), 847–849.
15. Johnstone A. H. and El-Banna H., (1986), Capacities, demands and processes: a predictive model for science education, Educ. Chem., 23, 80–84.
16. Johnstone A. H. and El-Banna H., (1989), Understanding learning difficulties – a predictive research model, Stud. High. Educ., 14(2), 159–168.
17. Lin H. S., Cheng H. J. and Lawrenz F., (2000), The assessment of students’ and teachers’ understanding of gas laws, J. Chem. Educ., 77, 235–238.
18. Löffler P., Pozas M. and Kauertz A., (2018), How do students coordinate context-based information and elements of their own knowledge? An analysis of students' context-based problem-solving in thermodynamics, Int. J. Sci. Educ., 40(16), 1935–1956.
19. Matijašević I., Korolija J. N., Ljuba M. and Mandić L. M., (2016), Translation of P = kT into a pictorial external representation by high school seniors, Chem. Educ. Res. Pract., 17, 656–674.
20. NGSS Lead States, (2013), Next generation science standards: For states, by states. Achieve, Inc. on behalf of the twenty-six states and partners that collaborated on the NGSS, Washington, DC: The National Academies Press.
21. Overton T. and Potter N., (2011), Investigating students’ success in solving and attitudes towards context-rich open-ended problems in chemistry, Chem. Educ. Res. Pract., 12, 294–302.
22. Overton T., Potter N. and Leng C., (2013), A study of approaches to solving open-ended problems in chemistry, Chem. Educ. Res. Pract., 14, 468–475.
23. Salta K. and Tzougraki C., (2012), Conceptual versus algorithmic problem-solving: focusing on problems dealing with conservation of matter in chemistry, Res. Sci. Educ., 41, 587–609.
24. Solaz-Portoles J. J. and Sanjose-Lopez V., (2009), Working memory in science problem solving: a review of research, Rev. Mex. de. Psi., 26(1), 79–90.
25. St Clair-Thompson H., Overton T. and Bugler M., (2012), Mental capacity and working memory in chemistry: algorithmic versus open-ended problem solving, Chem. Educ. Res. Pract., 13, 484–489.
26. Taasoobshirazi G. and Glynn S. M., (2009), College students solving chemistry problems: a theoretical model of expertise, J. Res. Sci. Teach., 46(10), 1070–1089.
27. Tekkumru-Kisa M., Stein M. K. and Coker R., (2017), Teachers’ learning to facilitate high-level student thinking: impact of a video-based professional development, J. Res. Sci. Teach., 55(4), 479–502.
28. Tsaparlis G., (2005), Non-algorithmic quantitative problem solving in university physical chemistry: a correlation study of the role of selective cognitive variables, Res. Sci. Tech. Educ., 23, 125–148.
29. Tumay H., (2014), Prospective chemistry teachers’ mental models of vapor pressure, Chem. Educ. Res. Pract., 15, 366–379.
30. Verhoeff R. P., Waarlo A. J. and Boersma K. T., (2008), Systems modelling and development of coherent understanding of cell biology, Int. J. Sci. Educ., 30, 543–568.
31. Wilensky U. and Reisman K., (2006), Thinking like a wolf, a sheep or a firefly: learning biology through constructing and testing computational theories – an embodied modelling approach, Cogni. Inst., 24, 171–209.
32. Zoller U. and Pushkin D., (2007), Matching higher-order cognitive skills (HOTS) promotion goals with problem-based laboratory practice in a freshman organic chemistry course, Chem. Educ. Res. Pract., 8(2), 153–171.

---