Development of problem-solving skills supported by metacognitive scaffolding: insights from students’ written work

Abstract

Despite problem solving being a core skill in chemistry, students often struggle to solve chemistry problems. This difficulty may arise from students trying to solve problems through memorising algorithms. Goldilocks Help serves as a problem-solving scaffold that supports students through structured problem solving and its elements, such as planning and evaluation. In this study, we investigated how first-year chemistry students solved problems, when taught with Goldilocks Help, and whether their problem-solving success and approaches changed over the course of one semester. The data comprised of student written problem-solving work, and was analysed using frequency analysis and grouped based on the problem-solving success and the extent of the demonstrated problem-solving elements. Throughout the course of semester, students exhibited increasingly consistent demonstration of structured problem solving. Nonetheless, they encountered difficulties in fully demonstrating such aspects of problem solving as understanding and evaluating concepts, which demand critical thinking and a firm grasp of chemistry principles. Overall, the study indicated progress in successful and structured problem solving, with a growing proportion of students demonstrating an exploratory approach as time progressed. These findings imply the need for incorporation of metacognitive problem-solving scaffolding, exposure to expert solutions, reflective assignments, and rubric-based feedback into wide teaching practice. Further research is required to extend the exploration of the effectiveness of metacognitive scaffolding, in particular via think-aloud interviews, which should help identify productive and unproductive uses of the problem-solving elements.

---

Introduction

Successful problem solving in chemistry necessitates students to possess conceptual knowledge and utilise metacognitive processes. However, students frequently encounter challenges when attempting to solve chemistry problems. One of the challenges relates to the inability to activate appropriate pre-knowledge (Gulacar et al., 2014). Some students may have a weak understanding of fundamental concepts, and prompting them to activate their conceptual knowledge becomes essential to facilitate problem solving. Chemistry problems can be overwhelming to students due to the extensive amount of information presented, and they may struggle with identifying and extracting necessary details to solve a problem (Bodner and McMillen, 1986). Extensive cognitive load (Sweller, 1994) can prevent students from formulating an effective solution strategy. Another obstacle to successful problem-solving is the lack of strategic approaches (De Corte et al., 2012). Problem solving in chemistry goes beyond applying algorithms or formulaic steps. It requires the ability to identify a solution strategy and implement it systematically. Without proper guidance, students may feel stuck, and struggle to progress in solving a problem.

Many students view chemistry problems as mere mathematical exercises (Randles and Overton, 2015; Phelps, 2019). They rely on their mathematical abilities and memorisation tactics to solve problems, overlooking the need for conceptual understanding. This perception creates a barrier to problem solving in chemistry, as solutions require both conceptual and quantitative competency (Towns et al., 2019). Students need to shift their mindset and approach chemistry problems holistically, recognising that conceptual reasoning is as crucial as mathematical skills. The focus on mathematical skills may stem from the emphasis placed on quantitative solutions in traditional chemistry courses (Phelps, 2019). Numeric answers are often seen as measures of rigour, leading students to prioritise obtaining numerical results over using and demonstrating conceptual reasoning. A sole focus on numerical results can detract from the deeper comprehension of chemistry concepts (Towns et al., 2019). To address this issue, practice and assessment should be designed to prompt students to explicitly demonstrate conceptual reasoning alongside their mathematical skills.

Mastery of conceptual reasoning as a measure of expertise in problem solving

Conceptual reasoning requires deliberate processing, which is one of the two (fast or slow) distinct modes of thinking (Evans, 2003; Kahneman, 2011; Varga and Hamburger, 2014). These modes share a common mechanism known as cognitive speed and cognitive effort, with control ranging from automatic to deliberate. Fast thinking is characterised by rapid, instinctive, and effortless processing (Varga and Hamburger, 2014). It can often be error-prone as it relies heavily on intuition and past experiences. People tend to engage in fast thinking when confronted with familiar tasks, where well-practised actions become the default for achieving their goals. However, fast thinking can hinder problem solving when it comes to novel tasks that require deliberate and reflective thinking. This particularly rings true for students learning to solve chemistry problems. While fast thinking can be productive for experts due to their extensive chemistry experience and well-developed schemas, it can be counterproductive for students (Rodriguez et al., 2019). Students may overlook metacognitive processes that cue them to reflect on misconceptions and potential errors, leading to “dead ends” and “false starts” (Yuriev et al., 2017).

As novice problem solvers, students should be encouraged to employ slow thinking to consciously control each step and treat sub-goals as individual tasks (Varga and Hamburger, 2014). Such deliberate approach to problem solving is characterised by effortful and conscious information processing (Rodriguez et al., 2019). With practice and experience, these sub-goals become integrated automatically, leading to faster and more efficient problem solving (Randles and Overton, 2015). Experts, thanks to their extensive experience, can perform required actions faster and with less conscious effort. Novices transitioning to experts will undergo this transformation through continued practice (Bodner and Herron, 2002).

Randles and Overton (2015) compared problem solving by undergraduate students, industry professionals (chemistry graduates who had at least ten years of experience working in the chemical industry), and academics. They found that undergraduates primarily focused on identifying needed information, using algorithms and calculations, and framing the problem. Their evaluation skills were limited. Industry professionals displayed similar approaches but engaged in more meaningful evaluation. The academic participants in that study, which were considered experts, utilised a wider range of approaches, including approximations, reflection, strategy development, and remained focused. They demonstrated a greater understanding of problem-solving and achieved higher success rates. Novices, such as undergraduates, can improve their problem-solving skills by incorporating evaluation throughout the process.

Metacognitive scaffolding for problem solving

To address student problem-solving challenges, it is important for instructors to intentionally model deliberate reflection and conceptual reasoning (Vo et al., 2022). By emphasising the importance of slow thinking and providing guidance on metacognitive processes, they can support their students in developing more effective problem-solving strategies and overcome the limitations of relying solely on fast thinking (Kahneman, 2011). Providing students with a problem-solving scaffold that emphasises metacognitive processes is one such approach to overcoming student problem-solving difficulties (Yuriev et al., 2017).

We have previously designed a metacognitive scaffold Goldilocks Help (GH) (Fig. 1) to support student problem solving (Yuriev et al., 2017). The scaffold involves five main elements. ‘Understand’ requires defining and deconstructing the problem. ‘Analyse’ and ‘Plan’ involve identifying knowns and unknowns, establishing relationships between them, and formulating a strategy of how these relationships could be used to determine the unknown(s). The ‘Implement’ element entails putting the plan into action, and “Evaluate’ requires checking and troubleshooting. This scaffold was developed following an extensive analysis of problem-solving workflows from a range of scientific disciplines (Yuriev et al., 2017).

Fig. 1 Goldilocks Help, a problem-solving workflow with metacognitive prompts. Reproduced from reference by Yuriev et al. (2017) with permission from The Royal Society of Chemistry.

It has been shown that scaffolding problem solving with the GH workflow resulted in a shift of students’ belief in their capacities to identify and use appropriate problem-solving strategies (Yuriev et al., 2017). When student written solutions were analysed, it was indeed found that students who demonstrated structured problem solving and explicit reasoning, were more successful in their problem-solving attempts. However, their written work still showed a lack of demonstrated monitoring and evaluation (Yuriev et al., 2019). To gain further insight into the successes and challenges of student engagement with structured problem solving, we have captured teaching associates’ (TAs’) perspectives on teaching with the GH scaffold (Vo et al., 2022), which allowed to identify barriers to students’ engagement with structured problem solving, as experienced and perceived by TAs. TA perspectives also included commentary on students engaging or not engaging with the ideas represented by the scaffold and on how well or poorly students performed in terms of problem solving (Vo et al., 2022). Thus, based on TA's views, we proposed four student profiles (Vo et al., 2022): a Sprinter was defined as a student that is not successful in problem solving and not using the scaffold, a Collector was classified as a student who utilises the scaffold without reaching problem-solving success, an Explorer was defined as a successful problem solver who actively uses the scaffold, and a Settler was categorised as a student who is a successful problem solver who no longer needs to use the scaffold. In order to confirm this proposed typology, we have suggested that the analysis of student-generated data is required (Vo et al., 2022). Student solutions could be used to ascertain to what extent students are demonstrating the ideas embedded in the scaffold, and student reflections should provide an insight into how students conceptualise these ideas and into why they do or do not engage with them.

Study rationale and research questions

In this study, we investigated student problem solving, when supported by metacognitive scaffolding, by collecting and analysing their written solutions and reflections. In the context of chemistry problem solving, meticulously demonstrating problem-solving processes is important for several educational and cognitive reasons. Firstly, explicit articulation of the problem-solving process helps in developing a better understanding and application of the underlying chemical principles and fosters deep reasoning. This practice aligns with the idea of cognitive apprenticeship, whereby learners engage in making their thinking visible, thus adopting and internalising expert metacognitive strategies through practice (Collins et al., 1991; Kirschner and Hendrick, 2024). Secondly, the process of carefully detailing each step in a solution encourages self-regulation, monitoring, and reflection, components of metacognition crucial for successful problem solving (Veenman et al., 2006). Through this reflective practice, students are able to evaluate the effectiveness of their strategies, identify errors, and adjust their approaches in future problems, thus improving their problem-solving efficacy over time.

In this paper, we detail the analysis of student written work and explore what processes were used to solve problems. On a whole cohort level, we investigate how students demonstrate specific elements/processes of structured problem solving and the progression of their abilities across one semester. We also analyse these two aspects (success and demonstrated problem-solving elements) on the individual student level, and again follow the dynamics of change over the semester. In a separate publication, we analyse the student written reflections to capture their perceptions on solving problems while engaging with the scaffold and applying structured problem solving (Vo et al., 2024a).

The research questions we address through the analysis of student written work are:

1. Do student problem-solving skills improve for cohorts taught/assisted with metacognitive scaffolding?

2. To what extent does individual student problem-solving work demonstrate their skill development, in terms of problem-solving success and demonstrated problem-solving elements?

Theoretical framework

This study incorporated three educational concepts to guide the data analysis and subsequent findings: scaffolding, metacognition, and self-regulated learning.

Scaffolding is an instructional approach that plays a crucial role in supporting learners, particularly novices, as they engage in complex learning tasks (Vygotsky, 1978; Belland, 2011). It serves as a bridge between a learner's current abilities and the desired learning outcomes, enabling them to gradually develop the necessary skills and knowledge to complete the task independently. In the context of problem-solving, scaffolding structures the learning process and prompts students to use metacognitive strategies, enabling them to solve problems unassisted (Wood et al., 1976; Reiser, 2004).

One key aspect of scaffolding is its ability to focus the learner's attention on the learning goals and essential aspects of the task (Wood et al., 1976). By providing clear objectives and highlighting relevant information, scaffolding helps students direct their cognitive resources towards the most important elements of the learning process. This targeted attention increases their awareness of potential knowledge gaps, allowing them to identify areas that require further exploration and understanding. In addition to focusing attention, scaffolding simplifies complex tasks by breaking them down into manageable components. By presenting information in a structured and organised manner, scaffolds make the learning process more accessible and less overwhelming, in particular but not exclusively, for cognitively overloaded learners (Sweller, 1988). This step-by-step approach enables learners to build a solid foundation of knowledge and skills, gradually increasing the complexity and challenge of the tasks as they progress.

A vital aspect of scaffolding is the use of prompts and guidance to support metacognitive and procedural thinking (Ge and Land, 2003). Prompts serve as cues or reminders that guide students' thinking processes and encourage them to engage in metacognitive reflection. These prompts can take the form of guiding-through questions, which stimulate rational and logical thinking, encourage students to articulate their thought processes, and help them recognise the need to evaluate the validity of their solutions.

A fundamental aspect of the scaffolding process is phasing out, or “fading”, of a scaffold (Belland, 2011). Fading involves the reduction of instructional support once the learner develops the grasp of the target skill. Identifying an appropriate time point for scaffold fading is a well-known challenge (Noyes and Cooper, 2019). This challenge is further complicated if a given student cohort significantly varies in terms of student abilities and self-regulatory capacities, necessitating adaptive scaffolding as a mechanism for calibrated support and individualised scaffolding (Azevedo and Hadwin 2005; Lieber et al., 2022a, 2022b).

Both hard and soft scaffolds can be employed, independently or in combination, to enhance meaningful learning experiences (Brush and Saye, 2002). Hard scaffolds are pre-planned and embedded within the learning materials, such as written materials or multimedia resources. They provide static support that addresses anticipated student difficulties and can be accessed by learners as they progress through the task. On the other hand, soft scaffolds, such as Socratic questioning (Elder and Paul, 1998), involve dynamic, situation-specific support provided by seasoned individuals. Soft scaffolding necessitates ongoing monitoring of student progress and timely adjustments to the instructional approach based on feedback and observations.

Metacognition involves actively controlling and regulating one's own cognitive processes during learning or problem solving (Garner and Alexander, 1989). It encompasses a set of activities such as strategic planning for approaching a learning task, monitoring one's comprehension and understanding, and evaluating the progress made towards completing the task. Through metacognition, individuals gain a deeper awareness and control over their thinking processes, enabling them to optimise their learning and problem-solving skills.

Metacognition comprises two main components: cognitive knowledge and cognitive regulation (Flavell, 1979; Schraw and Dennison, 1994). Cognitive knowledge refers to an individual's understanding of their own knowledge abilities and can be further divided into declarative (knowledge about one's skills, intellectual resources, and abilities as a learner), procedural (knowing how to implement learning procedures and strategies), and conditional (understanding when and why to use specific learning procedures). Cognitive regulation involves monitoring and controlling one's thought processes. It is achieved through strategies such as planning, to set goals and allocate resources, and information management, to effectively process information. Cognitive regulation also involves debugging strategies, to correct comprehension and performance errors, as well as monitoring and evaluation, to analyse performance and assess the chosen approaches.

Self-regulated learning (SRL) is characterised by students taking proactive measures to assume control over their own learning (Zimmerman and Pons, 1986). It is marked by personal initiative, perseverance, and adaptability (Zimmerman, 2008). Self-regulated learners understand their strengths and weaknesses, and seek relevant information to enhance their learning experience (Zimmerman, 1990). They exert control over their motivation, cognition, and behaviour. Metacognitive self-regulation plays a crucial role in SRL and involves planning, monitoring, and regulating the learning process. Planning activates prior knowledge and organises information, while monitoring integrates new information with existing knowledge. Regulatory activities, such as evaluation and checking, facilitate adjustments in problem-solving behaviour. Motivational processes encompass self-efficacy, and intrinsic task interest drives learners to invest effort. The behavioural process in SRL involves learners seeking advice, self-instructing, and reinforcing their learning. Self-regulated learners possess a deep awareness of the connection between self-regulatory strategies and learning outcomes, utilising these strategies to accomplish their goals and optimise their learning experience.

Methodology

Context and participants

In order to study student development of problem-solving skills, when taught with the metacognitive scaffold Goldilocks Help, we recruited first year undergraduate students studying a physical chemistry course in a Bachelor of Pharmaceutical Science at Monash University in 2020. Due to the COVID-19 pandemic, all instruction, learning activities, and assessment were done online. During Semester 1 (February–May), students studied four topics: (1) units and unit conversions, (2) acids and bases, (3) thermodynamics, and (4) chemical kinetics. Topic 1 included content familiar to students from high school, while the content in Topics 2–4 was novel to students.

This study was approved by Monash University Human Research Ethics Committee (Project ID: 23346). Convenience sampling was implemented (Koerber and McMichael, 2008). There were 178 students enrolled in the course who completed the course activities. The objectives of the study were explained to students, and their participation was optional. All students filled in an online consent form on Moodle, and 145 students provided permission to use their worked solutions in the study. Responses were de-identified, with names changed to participant numbers for anonymity. The cohort of recruited participants represented relevant qualities and experiences with respect to prior knowledge (i.e. high-school chemistry) and a specific level of chemistry mastery as stipulated by course requirements. These students completed prerequisite training in high school Mathematics and Chemistry. Additionally, a minimum Australian Tertiary Admission Rank (ATAR) of 84.3 was required to be eligible for the course. An ATAR is a number between 0.00 and 99.95 that indicates a student's position relative to all the students in their age group.

Scaffold implementation

The problem-solving scaffold, Goldilocks Help, was introduced to students through a combination of lectures and workshops. An induction workshop introduced students to structured problem solving to emphasise metacognition and self-regulation. In lectures, the scaffold was used to guide students through worked-example questions (Crippen and Brooks, 2009) using modelling instruction (Sweller, 2006; Van Gog et al., 2011). Both workshops and lectures constantly reinforced and demonstrated how to use the scaffold for solving chemistry problems. In workshops, students actively worked on problems using the Goldilocks Help to guide their solutions. TAs running the workshops were equipped with model solutions with explicit workflow prompts for all problem-solving elements. To assist students in monitoring their problem-solving processes, TAs encouraged students to make their thinking visible (Vo et al., 2022).

Data collection

For each of the four topics (Table 1), students were given a two-part problem-solving activity (Fig. 2). Task 1 of the activity required students to solve an allocated problem and to self-assess their problem-solving abilities. Analysis of this self-assessment will be published separately (Vo et al., 2024b). Students were provided with the problem-solving rubric (Table 2) prior to the activity submission and were encouraged to use the Goldilocks Help scaffold as a guide while solving the problem. After the submission deadline for Task 1, students received the expert solution to the problem. Task 2 of the activity required students to submit a reflection that compared their solution to an expert solution (comparative reflection). Analysis of these reflections will be published separately (Vo et al., 2024a). Students were also required to again self-assess their problem-solving abilities in light of the new-found knowledge of the expert solution.

Table 1 The number of collected student solutions for the four topics

	Number of submissions
Topic 1: unit conversions	130
Topic 2: acids and bases	134
Topic 3: thermodynamics	131
Topic 4: chemical kinetics	135
Total	530

---

Fig. 2 Two-part problem-solving activity with a focus on Task 1 (solving a problem).

Table 2 Assessment rubric for the problem-solving activity^a

Criteria	Level
	Full	Partial	Insufficient
a For each topic, the problem-solving rubric was elaborated to include elements, specific for each problem (Appendix 2, ESI).
Necessary information (understanding problem statement)	Identifies all key information, variables, etc. needed to solve the problem	Identifies some key information, variables, etc. needed to solve the problem	Identifies little key information, variables, etc. needed to solve the problem
Evidence of thought process (analysis)	Evidence of thought process is present, organised and complete	Evidence of thought process is present but is incomplete	Evidence of thought process is present but is disjointed
Strategy (planning)	Arrives at a strategy that is functional and optimal	Arrives at a strategy that is functional but not ideal	Arrives at a strategy that is not functional and shows little understanding of the task
Execution and completeness (implementation)	Uses all of the necessary information correctly and addresses all parts of the problem/task	Uses some of the necessary information correctly and addresses some parts of the problem/task	Uses little of the necessary information correctly and addressed few parts of the problem/task
Judgement of reasonableness of solution (evaluation)	Rationale is relevant and correct	Rationale is relevant but not correct	Rationale is not relevant

---

This study explores the first task of the 2-part activity and focuses on the evaluation of student solutions. All problems can be found in Appendix 1, ESI.† Student solutions ranged between 1–2 pages. Fig. 3 is an example of a student solution. A total of 530 student solutions were collected.

Fig. 3 An illustration of a student solution for Topic 4 problem.

Data analysis

In this study we used a descriptive quantitative approach to analyse and interpret the data collected (Bloomfield and Fisher, 2019). Initially, student solutions were assessed to determine whether the numeric answer was correct, categorising their problem-solving outcomes as “successful” or “unsuccessful”. Following this categorisation, their solutions were scored using the problem-solving rubric (Table 2). Each problem-solving element was scored none/insufficient, partial, or full. The rubric was developed to reflect the expectations for structured problem solving captured in the Goldilocks Help scaffold (Yuriev et al., 2017) and skills targeted in ELIPSS problem-solving rubric (Cole et al., 2017; Cole et al., 2018). The analysis was performed by two researchers (KV and EY). For each topic, the researchers initially coded 20% of the solutions independently, after which they met to compare and refine their scoring, until full agreement was reached. The first author then scored the remainder of the solutions. At this point, the two researchers met again, reviewed any solutions that required additional discussion, and checked for the consistency of rubric application throughout the total solution set.

Once all solutions were assessed and scored, the data were analysed on two levels: for the cohort as a whole and for each student individually.

For the whole cohort, a frequency analysis was conducted to investigate the problem-solving processes demonstrated in student solutions. The problem-solving elements (understand, analyse, plan, implement, and evaluate) and the three levels of demonstration (none, partial, or full) were totalled. The total frequency of each problem-solving element was standardised into percentage form (compared the number of students demonstrating each problem-solving element relative to the total number of students being examined e.g. by topic or by participant type: successful, unsuccessful, all) and visualised as stacked columns (Fig. 4 and 5).

Fig. 4 The percentage of student use (full, partial, none) of problem-solving elements (U: understand, A: analyse, P: plan, I: implement, E: evaluate) in student solutions (Topic 1: unit conversions, Topic 2: acids and bases, Topic 3: thermodynamics, and Topic 4: chemical kinetics).

Fig. 5 The percentage of student use (full, partial, none) of problem-solving elements (U: understand, A: analyse, P: plan, I: implement, E: evaluate) in successful and unsuccessful student solutions (Topic 1: unit conversions, Topic 2: acids and bases, Topic 3: thermodynamics, and Topic 4: chemical kinetics).

For the individual analysis, students were grouped based on their problem-solving success and the extent of the demonstrated problem-solving elements. The demonstration of each of the five problem-solving elements was scored from 0 (none/insufficient) to 0.5 (partial) to 1 (full), giving a total possible rubric score of 5. A total score below 2.5 was classified as none or limited demonstration, and a total score of 2.5 and above was classified as medium to substantial demonstration. Four resulting groups were: incorrect answer/low demonstration; incorrect answer/high demonstration; correct answer/low demonstrated; correct answer/high demonstration. To capture a longitudinal perspective on the development of problem-solving skills, student groups were examined across all four topics across a semester. The dynamics of problem-solving skill development was followed by tracking the change in student groups and illustrated with alluvial diagrams, constructed using SankeyMATIC (Bogart, 2014).

Results & discussion

Demonstration of structured problem solving in student written work (RQ1: cohort-level data)

In their solutions, students were expected to deconstruct the problem statement for explicit and implicit information to understand the problem, analyse the problem by identifying the relationships between relevant variables, implement a planned solution with dimensional analysis, and evaluate their answer. Through frequency analysis, an insight into student application of specific elements of structured problem solving, and the progression of their abilities across one semester were captured. The cohort of students demonstrated the problem-solving elements more consistently as the semester progressed (Fig. 4). For example, in Topic 1, 21% of students showed full Analysis of the problem, with that fraction increasing to 77% by the end of the semester (Topic 4). However, demonstrating Understanding and Evaluation was more challenging for students to master. For example, in Topic 2, only a small fraction of students fully demonstrated these elements (39% and 15%, respectively), compared to majority of students doing that for Analysis and Implementation (64% and 72%, respectively).

Understanding and Evaluation are likely more challenging due to requiring deeper reasoning, better language skills, and understanding of concepts (Yuriev et al., 2017; Vo et al., 2022). Students' reasoning abilities can be impeded by a multitude of factors. Firstly, students often exhibit a preference for fast thinking (Kahneman, 2011). Though characterised as quick and intuitive, fast thinking can often lead to a lack of deep reasoning for inexperienced problem solvers. In order for students to be successful problem solvers, a shift towards slow thinking is required (Rodriguez et al., 2019). Slowing down and engaging in deliberate reasoning allows students to consider multiple perspectives, identify implicit features of a problem, and develop more comprehensive solutions. By slowing down, they can reflect on their implicit biases and misconceptions to avoid incorrect assumptions, and overlook important information necessary for solving the problem.

Moreover, there is an overemphasis on finding the ‘right’ equation and carrying out mathematical calculations, rather than thinking through the problem holistically (Bodner and Herron, 2002). This behaviour was also described in Rodriquez and co-workers’ work on chemical kinetics (2019), noting that students should shift away from plugging values into the first equation that comes to mind as their primary approach. Such an approach limits students’ ability to engage in conceptual reasoning and understanding of the underlying principles, also seen in Phelps’ research (2019). The reliance on equations may stem from students not fully realising the importance of explaining their reasoning. This lack of awareness can result in calculation mistakes and difficulty in troubleshooting errors. Consequently, they may encounter false starts, dead-ends, and incomplete strategies (Yuriev et al., 2017). To address these challenges, students’ problem solving needs to be supported by scaffolding with metacognitive and instructional prompts and by providing opportunities for deliberate self-regulating behaviour, involving planning, monitoring, and evaluating their thinking and learning process (Graulich et al., 2021). By developing these skills, students can enhance their reasoning abilities and improve their problem-solving performance.

The frequency analysis also showed that successful students demonstrated problem-solving elements more consistently (Fig. 5). For example, in Topic 2, 81% of successful students fully demonstrated understanding the problem statement, whereas only 10% of unsuccessful students did, with 81% of unsuccessful students not demonstrating understanding at all. These results suggest that solving the problem unsuccessfully was likely due to a lack of understanding of the problem statement as well as the lack of use of other problem-solving processes that were prevalently used by successful students. As asserted by Schraw and Dennison (1994) and Zimmerman and Pons (1986), self-regulated and metacognitively aware learners demonstrate greater strategic thinking and achieve better performance compared to unaware and un-regulated learners. By being conscious of their own learning processes, strength and shortcomings, students are able to exert control over their motivations, cognition, and behaviours to solve problems successfully.

Over the duration of the semester, the overall cohort of students demonstrated problem-solving elements more consistently and were getting more successful (Fig. 5). The number of successful students increased from 54 to 94 to 115 (in “novel” content Topics 2 to 4). The consistent exposure to Goldilocks Help enabled students to become familiar with incorporating metacognitive processes in their problem solving. This fostered a gradual transfer of responsibility of metacognitive prompting from the instructor to the learner, empowering students to become self-regulated learners.

Dynamics of skill development, in terms of problem-solving success and demonstrated problem-solving processes (RQ2: individual student-level data)

The categorisation of each student into one of the four groups was based on their problem-solving outcome (successful or unsuccessful) and problem-solving elements demonstrated in their written work (none/limited or medium/substantial).

A significant number of students, 33%, fell into the correct answer/low demonstration group, when they dealt with a relatively simple Topic 1 problem in a familiar content area (Fig. 6; Green). When presented with a routine exercise, these students were able to solve the problem, likely without requiring scaffold assistance, and therefore did not show their work in detail, which resulted in low solution scores. However, when faced with the novel content (Topics 2–4), only 5%, 8%, and 5% of students, respectively, fell into this group. The further discussion focuses on the dynamics of student groups between novel-content Topics, from 2 to 4.

Fig. 6 Percentage of students based on student groups in the four topics: Violet, incorrect answer/low demonstration; Orange, incorrect answer/high demonstration; Green, correct answer/low demonstration; Blue, correct answer/high demonstration; Grey, no submission.

In this study, the percentage of students appearing in the incorrect answer/low demonstration group (Fig. 6; Violet) decreased from 39% to 10% from Topic 2 to Topic 4. This reduction occurred largely due to students progressing through to the incorrect answer/high demonstration and correct answer/high demonstration groups (Fig. 6; Orange and Blue, respectively). For example, 8% and 17% of students progressed to these two groups, respectively, between Topics 2 and 3 (Fig. 7). And 6% of students progressed to the correct answer/high demonstration group between Topics 3 and 4 (Fig. 7).

Fig. 7 Sankey diagram of all student group transitions in all four topics in the semester. Colour coding: Violet, incorrect answer/low demonstration; Orange, incorrect answer/high demonstration; Green, correct answer/low demonstration; Blue, correct answer/high demonstration; Grey, no submission. Topic 1 (Yellow line) – familiar content; Topics 2–4 (Red line) – novel content.

Similarly, the percentage of students appearing in the incorrect answer/high demonstration group (Fig. 6; Orange) also reduced throughout the semester, specifically, from 19% to 5% from Topic 2 to Topic 4. A majority of these students progressed to the correct answer/high demonstration group (Fig. 6; Blue): 11% of students progressed between Topics 2 and 3, and 14% between Topics 3 and 4 (Fig. 7).

Notably, the correct answer/high demonstration group experienced the largest influx of transitions into the group from Topic 2 to Topic 4 (Fig. 6 and 7; Blue). In addition to this, a large majority of students appearing in this group in Topic 2 remained in the group.

Given more time and continued exposure to the scaffold, we predict to see the dynamic shift from correct answer/high demonstration to correct answer/low demonstration group. This transition is related to the internalisation of the scaffold (Belland, 2011). Once structured problem-solving skills are developed, and naturally incorporated into their thought processes, students are expected to no longer require the explicit use of the scaffold. The expertise-reversal effect suggests that for experienced problem solvers, the scaffold may hinder rather than benefit their progress (Kalyuga et al., 2003). Therefore, scaffold should be viewed as a temporary tool that is introduced to assist students in the development of cognitive processes. Once a student is able to handle challenging problems without assistance, the scaffold can be phased out, or faded (Belland, 2011). Instructors often need to make difficult decisions about when scaffolding should be faded (Noyes and Cooper, 2019). Being able to recognise a student(s) as belonging to the correct answer/low demonstration group should help instructors in identifying an appropriate learning point where scaffolding could be withdrawn.

To summarise, within a period of one semester, students displayed an improvement in successful and structured problem solving (Fig. 8). We observed a large influx into the correct answer/high demonstration group that is characterised with success and high degree of demonstrated problem-solving processes and reasoning, but requiring more practice to internalise the scaffold. Understanding these groups can help educators tailor their instruction to meet the diverse needs of students.

Fig. 8 Sankey diagram of transitions of student groups across the semester from Topic 2 to Topic 4 (novel content only): Violet, incorrect answer/low demonstration; Orange, incorrect answer/high demonstration; Green, correct answer/low demonstration; Blue, correct answer/high demonstration; Grey, no submission.

Conclusion and implications

Goldilocks Help scaffold was used to facilitate the development of metacognitive skills and to empower students to become self-regulated learners. The findings of this study revealed important insights into the dynamics of student adoption of problem-solving processes, embedded in the scaffold, over the course of one semester. Overall, the study showed improvement in successful and structured problem-solving. As the semester progressed, the cohort of students demonstrated the problem-solving elements more consistently. However, students faced challenges in fully demonstrating the Understanding and Evaluation elements, which require critical thinking and a strong grasp on chemistry concepts. The preference for fast thinking hindered their ability to engage in deliberate and conscious reasoning processes, resulting in false starts and dead-ends. Successful students consistently demonstrated the problem-solving elements and displayed a greater understanding of the problem statement. The results highlighted the importance of metacognitive awareness and self-regulation in problem solving, with students who exhibited these qualities demonstrating better academic performance. Over the course of the semester, the overall cohort of students showed improvement in their problem-solving abilities.

In this study, we also categorised students into one of the four groups based on their problem-solving outcome and extent of demonstrated problem-solving elements. Over the course of the semester, the percentage of successful students who did not displayed detailed problem-solving work was stable, while the proportion of students who explicitly demonstrated problem-solving processes, embedded in the scaffold, increased. We predict that the latter group of students would eventually internalise the scaffold. This change would indicate reaching conditions for scaffold fading (Belland, 2011).

Limitations and research implications

In this study, each student was categorised into one of four groups based on their problem-solving outcome and the extent to which they demonstrated reasoning in their solutions. We hypothesise that these four groups may align, fully or partially, with the four problem-solving profiles we previously proposed: Sprinter, Collector, Explorer, and Settler (Vo et al., 2022). The analysis presented in the current work can go only as far as making claims about what students do when they are solving problems, but not the reasons or motivations for students’ actions. Establishing such reasons and motivations requires analysis of their perceptions, which are not discernible from the written problem-solving work alone. Students’ perceptions could be investigated using students’ comparative reflections, collected as described in the Methodology section, and used to get an insight into possible cognitive, emotional, and behavioural factors that could contribute to students’ views on and praxis of problem solving. These perceptions are investigated in a separate manuscript (Vo et al., 2024a).

We explored student development of structured problem solving over a period of one semester only. While the findings demonstrated the promising improvement in successful and structured problem solving, we did not identify an appropriate time point for scaffold fading. In order to rigorously examine the fading of scaffolding, a comprehensive longitudinal perspective is required to understand its long-term impact on student problem-solving skills. A mixed-methods approach could be used to investigate fading, which would include quantitative data on the frequency and types of scaffolding interactions and qualitative data, such as that obtained from classroom observations or think-aloud interviews.

The data analysis presented here invariably involved a binary system of correctness, i.e. the successful/unsuccessful dimension, which means that a simple error such as a calculator input mistyping could move a student vertically between groups. In this context, it is important to remember that the scaffold includes an evaluation aspect, which should encourage students to check their answer hence minimising these types of errors. Overall, while similar mistakes may in principle place students into the bottom row groups, such individual occasions are not expected to significantly affect the findings on the whole cohort level, over the course of four activity cycles.

Due to ethical considerations, the study did not explore uncontrolled independent variables like prior academic ability. As a consequence, the findings may not be widely applicable or generalisable to other educational contexts.

Teaching implications

Students often encounter difficulties when it comes to evaluating their answers, due to a lack of skills, or being unaware or uncertain about the appropriate problem-solving methods to use. To address the persistent struggles with evaluation-based problem-solving strategies, it is essential for classroom practices and instructional materials to go beyond mere content recall and procedural application. Students should be required to actively demonstrate their reasoning and critical thinking skills. Emphasising metacognitive strategies and integrating metacognitive training with content teaching can play a crucial role. Our research group attempts to do so through the incorporation of Goldilocks Help in our teaching, through modelling instruction in lectures (Yuriev et al., 2017) and continuous reinforcement of the scaffold during problem-solving sessions (Vo et al., 2022). The scaffold offers prompts and exemplar suggestions that encourage students to engage in thorough reasoning during problem solving. By incorporating a combination of Goldilocks Help problem-solving scaffold, exposure to expert solutions, reflective assignments, and rubric-based feedback, the likelihood of students adopting effective evaluation strategies is significantly increased.

In terms of instructor training, particularly for early career instructors and teaching associates, it is crucial to address the diverse characteristics of students. Instructors should be equipped with the skills to recognise student preparedness, attitudes, and the need for additional support (Piontkivska, Gassensmith and Gallardo-Williams, 2021). Both instructors and students should be encouraged to refer back to the scaffold when they encounter difficulties during problem solving, as it can provide a common language to communicate between the instructor and student or between peers (Vo et al., 2022). Additionally, instructors should continuously demonstrate the benefits of the scaffold in problem solving as it will facilitate student engagement with the scaffold.

Finally, due to the nature of a metacognitive scaffold such as Goldilocks Help, it cannot be effectively withdrawn since it is not specific for a given task but rather structures problem solving as a process. Therefore, instructors need to be able to identify when students have internalised the scaffold and no longer require additional prompting or guidance. In our previous investigation, the TAs described such a state as a “habitual” structured problem solving (Vo et al., 2022). We propose that grouping students in terms of problem-solving success and explicitness of their reasoning offers a mechanism for identifying an appropriate point for scaffold fading on a student-by-student basis, which is both inclusive and individualised. Once instructors determine that a student or a student group no longer requires comprehensive scaffolding, they should limit their support to providing only limited hints, refinements, and feedback. The advice should shift from explicit instructions (“Write down the step to achieve [outcome]”) to more open-ended suggestions that encourage students to generate their own strategies (“What could be your approach here?”). For such challenging aspects of problem solving as Evaluation, the scaffold-faded task design should include removing the explicit “Justify your answer” final prompt. As students demonstrate increased competence, the frequency of support can be adjusted and ultimately reduced, directions can be provided only when students encounter difficulties or when they seek help, fostering greater autonomy and self-reliance in their problem-solving process. Thus, continuous evaluation and customised fading/adaptive scaffolding should account for individual learning progress, ensure that effective support is phased out appropriately, and prepare students for independent problem solving.

Data availability

Data collected from human participants, analysed to generate data depicted in Fig. 4–8, are not available for ethical confidentiality reasons. The data supporting Fig. 4–8 have been included as part of the ESI.†

Conflicts of interest

There are no conflicts of interest to declare.

Acknowledgements

The authors are grateful to Prof. Renée Cole for sharing her expertise in assessment of process skills. KV acknowledges the receipt of RTP (Research Training Program) Postgraduate Scholarship. We are also grateful to participants for providing access to their written work analysed in this study.

References

1. Azevedo R. and Hadwin A. F., (2005), Scaffolding self-regulated learning and metacognition – Implications for the design of computer-based scaffolds, Instr. Sci., 33, 367–379.
2. Belland B. R., (2011), Distributed cognition as a lens to understand the effects of scaffolds: the role of transfer of responsibility, Educ. Psychol. Rev., 23, 577–600.
3. Bloomfield J. and Fisher M. J., (2019), Quantitative research design, J. Australasian Rehabilitation Nurses Association, 22, 27–30.
4. Bodner G. M. and Herron J. D., (2002), Problem-solving in chemistry, in Chemistry education: towards research-based practice, J. K. Gilbert, O. DeJong, R. T. Justi, D. Treagust and J. Driel (ed.), Kluwer Academic Publishers, ch. 11, pp. 235–266.
5. Bodner G. M. and McMillen T. L. B., (1986), Cognitive restructuring as an early stage in problem solving, J. Res. Sci. Teach., 23, 727–737.
6. Bogart S., (2014), SankeyMATIC, https://sankeymatic.com (Accessed December 17, 2023).
7. Brush T. A. and Saye J. W., (2002), A summary of research exploring hard and soft scaffolding for teachers and students using a multimedia supported learning environment, J. Interact. Online Learn., 1, 1–12.
8. Cole R., Lantz J., Ruder S., Reynders G. and Stanford C., (2018), Enhancing learning by assessing more than content knowledge, Salt Lake City, Utah: Conference proceeding presented at the 2018 ASEE Annual Conference & Exposition, https://peer.asee.org/29991.
9. Cole R., Lantz J., Ruder S. and Stanford C., (2017), Enhancing learning by improving process skills in STEM, https://elipss.com (Accessed December 17, 2023).
10. Collins A., Brown J. S. and Holum A., (1991), Cognitive apprenticeship: making thinking visible, Am. Educ., 15, 6–11, 38–46, https://www.aft.org/ae/winter1991/collins_brown_holum.
11. Crippen K. J. and Brooks D. W., (2009), Applying cognitive theory to chemistry instruction: the case for worked examples, Chem. Educ. Res. Pract., 10, 35–41.
12. De Corte E., Verschaffel L. and Van Dooren W., (2012), Heuristics and problem solving, in Encyclopedia of the Sciences of Learning, N. M. Seel (ed.), Boston MA: Springer US, pp. 1421–1424.
13. Elder L. and Paul R., (1998), The role of Socratic questioning in thinking, teaching, and learning, The Clearing House, 71, 297–301.
14. Evans J. S. B., (2003), In two minds: dual-process accounts of reasoning, Trends Cogn. Sci., 7, 454–459.
15. Flavell J. H., (1979), Metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry, Am. Psychol., 34, 906–911.
16. Garner R. and Alexander P. A., (1989), Metacognition: answered and unanswered questions, Educ. Psychol., 24, 143–158.
17. Ge X. and Land S. M., (2003), Scaffolding students' problem-solving processes in an ill-structured task using questionprompts and peer interactions, Educ. Technol. Res. Dev., 51, 21–38.
18. Graulich N., Langner A., Vo K. and Yuriev E., (2021), Scaffolding metacognition and resource activation during problem solving: a continuum perspective, in Problems and problem solving in chemistry education, G. Tsaparlis (ed.), Royal Society of Chemistry, pp. 38–67.
19. Gulacar O., Eilks I. and Bowman C. R., (2014), Differences in General Cognitive Abilities and Domain-Specific Skills of Higher- and Lower-Achieving Students in Stoichiometry, J. Chem. Educ., 91, 961–968.
20. Kahneman D., (2011), Thinking, fast and slow, New York: Farrar, Straus and Giroux.
21. Kalyuga S., Ayres P., Chandler P. and Sweller J., (2003), The expertise reversal effect, Educ. Psychol., 38, 23–31.
22. Kirschner P. A. and Hendrick C., (2024), How Learning Happens: Seminal Works in Educational Psychology and What They Mean in Practice, 2nd edn, Routledge, ‘Making thinking visible’, ch. 29, pp. 310–318.
23. Koerber A. and McMichael L., (2008), Qualitative sampling methods: a primer for technical communicators, J. Bus. Tech. Commun., 22, 454–473.
24. Lieber L. S., Ibraj K., Caspari-Gnann I. and Graulich N., (2022a), Closing the gap of organic chemistry students' performance with an adaptive scaffold for argumentation patterns, Chem. Educ. Res. Pract., 23, 811–828.
25. Lieber L. S., Ibraj K., Caspari-Gnann I. and Graulich N., (2022b), Students' individual needs matter: a training to adaptively address students' argumentation skills in organic chemistry, J. Chem. Educ., 99, 2754–2761.
26. Noyes K. and Cooper M. M., (2019), Investigating student understanding of london dispersion forces: a longitudinal study, J. Chem. Educ., 96, 1821–1832.
27. Phelps A. J., (2019), “But you didn’t give me the formula!” and other math challenges in the context of a chemistry course, in It's Just Math: Research on Students’ Understanding of Chemistry and Mathematics, M. H. Towns, K. Bain and J.-M. G. Rodriguez (ed.), ACS Publications, ch. 7, pp. 105–118.
28. Piontkivska H., Gassensmith J. J. and Gallardo-Williams M. T., (2021), Expanding inclusivity with learner-generated study aids in three different science courses, J. Chem. Educ., 98, 3379–3383.
29. Randles C. A. and Overton T. L., (2015), Expert vs. novice: approaches used by chemists when solving open-ended problems, Chem. Educ. Res. Pract., 16, 811–823.
30. Reiser B. J., (2004), Scaffolding complex learning: the mechanisms of structuring and problematizing student work, J. Learn. Sci., 13, 273–304.
31. Rodriguez J. G., Bain K., Hux N. P. and Towns M. H., (2019), Productive features of problem solving in chemical kinetics: more than just algorithmic manipulation of variables, Chem. Educ. Res. Pract., 20, 175–186.
32. Schraw G. and Dennison R. S., (1994), Assessing metacognitive awareness, Contemp. Educ. Psychol., 19, 460–475.
33. Sweller J., (1988), Cognitive load during problem solving: effects on learning, Cogn. Sci., 12, 257–285.
34. Sweller J., (1994), Cognitive load theory, learning difficulty, and instructional design, Learn. Instr., 4, 295–312.
35. Sweller J., (2006), The worked example effect and human cognition, Learn. Instr., 16, 165–169.
36. Towns M. H., Bain K. and Rodriguez J.-M. G., (2019), How did we get here? Using and applying mathematics in chemistry in It's Just Math: Research on Students' Understanding of Chemistry and Mathematics, M. H. Towns, K. Bain and J.-M. G. Rodriguez (ed.), ACS Publications, ch. 1, pp. 1–8.
37. Van Gog T., Kester L. and Paas F., (2011), Effects of worked examples, example-problem, and problem-example pairs on novices’ learning, Contemp. Educ. Psychol., 36, 212–218.
38. Varga A. L. and Hamburger K., (2014), Beyond type 1 vs. type 2 processing: the tri-dimensional way, Front. Psychol., 5, art.993 DOI:10.3389/fpsyg.2014.00993.
39. Veenman M. V. J., Van Hout-Wolters B. H. A. M. and Afflerbach P., (2006). Metacognition and learning: conceptual and methodological consideration, Metacogn. Learn., 1, 3–14.
40. Vo K., Sarkar M., White P. J. and Yuriev E., (2022), Problem solving in chemistry supported by metacognitive scaffolding: teaching associates’ perspectives and practices, Chem. Educ. Res. Pract., 23, 436–451.
41. Vo K., Sarkar M., White P. J. and Yuriev E., (2024a), Metacognitive problem solving: exploration of students’ perspectives through the lens of multi-dimensional engagement, (manuscript submitted to Chem. Educ. Res. Pract.).
42. Vo K., Sarkar M., White P. J. and Yuriev E., (2024b), Integrating metacognitive training, reflective comparative benchmarking, self-assessment, and internal feedback generation for problem-solving skill development, manuscript in preparation.
43. Vygotsky L., (1978), Interaction between learning and development, in Mind in society the development of higher psychological processes, M. Cole, V. John-Steiner, S. Scribner and E. Souberman (ed.), Cambridge, MA: Harvard University Press, pp. 79–91.
44. Wood D., Bruner J. S. and Ross G., (1976), The role of tutoring in problem solving, J. Child Psychol. Psychiatry, 17, 89–100.
45. Yuriev E., Basal S. and Vo K., (2019), Developing problem-solving skills in physical chemistry, in Teaching chemistry in higher education: A festschrift in honour of Professor Tina Overton, M. K. Seery and C. McDonnell (ed.), Dublin: Creathach Press, pp. 55–76.
46. Yuriev E., Naidu S., Schembri L. and Short J., (2017), Scaffolding the development of problem-solving skills in chemistry: guiding novice students out of dead ends and false starts, Chem. Educ. Res. Pract., 18, 486–504.
47. Zimmerman B. J., (1990), Self-regulated learning and academic achievement: an overview, Educ. Psychol., 25, 3–17.
48. Zimmerman B. J., (2008), Investigating self-regulation and motivation: historical background, methodological developments, and future prospects, Am. Educ. Res. J., 45, 166–183.
49. Zimmerman B. J. and Pons M. M., (1986), Development of a structured interview for assessing student use of self-regulated learning strategies, Am. Educ. Res. J., 23, 614–628.

---

Footnote

† Electronic supplementary information (ESI) available See DOI: https://doi.org/10.1039/d3rp00284e

---