Tina Overton*, Nicholas Potter and Christopher Leng
University of Hull, Cottingham Road, Hull, HU6 7RX, UK. E-mail: t.l.overton@hull.ac.uk
First published on 7th June 2013
This paper describes the outcomes of a qualitative investigation into the range of different approaches that students use to solve open-ended, context rich problems. The study involved a small cohort of students individually solving open-ended, context-rich problems using a think aloud protocol. The problems required the students to develop a strategy, to identify data required and to make estimations. The problems did not lead to a single correct answer but rather a range of acceptable answers. Analysis of the transcripts and recordings of the sessions resulted in a limited number of categories of approaches to solving the problems. Applications of these approaches to individual students showed that there were three different types of problem-solver, described here as novice, expert and transitional. The results provide insight that will help tutors change how they, and subsequent students, approach problem solving.
Problem solving has been defined as ‘what you do, when you don't know what to do’ (Wheatley, 1984). Anderson (2005) described problem solving as ‘any goal-directed sequence of cognitive operations’. The types of problems set in examinations or assessments in higher education chemistry are largely algorithmic (Bennett, 2008; Pappa and Tsaparlis, 2011). Algorithmic problems use mainly lower order cognitive skills (LOCS), whereas more open ended problems call upon higher order cognitive skills (HOCS) (Zoller and Pushkin, 2007). Problems requiring the application of HOCS demand more than just knowledge to reach a solution. Knowledge or known theory must be applied in unfamiliar contexts using such skills as analysis, connection making, synthesis and critical thinking. Bodner (1986) suggest that the difference between an exercise and a problem is the result of differences in the level of familiarity with similar tasks the individual brings to a given task. Funke (2010) describes problem solving as an activity that requires more than routine action and thinking and states that complex cognition is required to solve complex problems. Funke views complex problem solving as involving intransparency, dynamics (underlying psychological forces, such as motivation and emotion), or polytely (multiple simultaneous goals).
More than 50 years ago, Polya proposed a model of problem solving that consists of four steps or stages (Polya, 1945); understand the problem, devise a plan, carry out the plan, look back. Bodner and Domin (1991) reports a much less structured approach to solving problems and identified many more steps. His model consisted of the following steps; read the problem, read the problem again, write down what you hope is the relevant information, draw a picture, make a list, or write an equation or formula to help you begin to understand the problem, try something, try something else, see where this gets you, read the problem again, try something else, see where this gets you, test intermediate results, read the problem again, write down ‘an’ answer, test the answer to see if it makes sense. This is obviously a more iterative and reflective approach to that proposed by Polya and the difference between them may reflect the difference between expert and novice problem solvers.
Developing problem solving skills has been the subject of much research into science education (Zoller et al., 1999; Dori and Hameiri, 2003). Gabel and Bunce (1994) proposed that the factors affecting students' success in problem solving are threefold: the type of problem and the underlying concepts of the problem; the learner characteristics (including cognitive styles and knowledge base); learning environment factors such as individual or group activity. Bunce attempted to enhance chemistry students' skill in problem solving through problem categorization. Students were taught how to solve problems using the explicit method of problem solving (EMPS) (Bunce and Heikkinen, 1986). The EMPS involves encoding the information given in a problem, relating this to what is already known, and planning a solution. This study showed that training in categorization skills and the EMPS can lead to higher achievement in problem solving. However, there was no link between understanding of the chemical concepts involved and the problem-solving skills ability.
Johnstone and El-Banna (1986) stressed the importance of working memory capacity in learning science and Reid (2009) suggested that problems that place a high load on working memory may be a better measure of a student's working memory capacity than of problem solving ability. Tsaparlis (2005) investigated the effect of several cognitive variables on students' ability to solve open-ended problems in physical chemistry. The results showed a strong correlation between M-capacity and disembedding ability and problem solving scores. In a study by St Clair-Thompson et al. (2012) it was found that the best predictors of algorithmic and open-ended problem solving abilities were tests that measured different elements of working memory. Performance on algorithmic problems was predicted by the counting recall tests for working memory which utilises lower order cognitive skills. Performance on open-ended problems was predicted by the figural intersection test and backwards digit recall test. These which measure mental capacity which utilises higher order cognitive skills. These results demonstrated that different cognitive resources are required for solving algorithmic and open-ended problems. In a study of online problem solving, Bühner et al. (2008) found that the working memory component of intelligence is the relevant aspect in predicting problem-solving ability. The nature of the task, its verbal, figural, or numerical content, might play an important role in explaining this relationship. When a task requires information to be stored and processed as well as integrated with elements held in memory, the aspects of intelligence that are related to working memory become most influential.
Kapa (2007) studied the effect of providing metacognitive support mechanisms (MSM) to students during their problem solving activities. MSMs are verbal prompts such as “What are you asked to find?” “What is given in the problem?” The study showed that these prompts were effective in helping students transfer from solving structured, closed problems to open-ended problems. Cooper (2009) studied the influence of students' metacognitive behaviours on their ability to solve ill defined problems in chemistry. Using an automated online instrument that collected and analysed students actions, a significant correlation was found between the use of metacognitive strategies and the complexity of the problems students could handle effectively. Scherer and Tiemann (2012) investigated the relationship between problem solving ability and metacognitive strategy knowledge within an online environment. Their subjects were solving analytical, or closed, problems in which all the required data are provided, and complex problems, in which all required information is not provided. The study showed substantial correlation between strategy knowledge and analytical problem solving, but not with complex problem solving. The authors report that the ability to solve analytical problems can be distinguished from the ability to solve complex problems and that fostering problem-solving skills has to move beyond the development of strategy knowledge. Chen and Wu (2012) also found that metacognitive strategies did not lead to higher performances with complex problems in online environments. In a study of chemistry students' ability to solve quantitative, closed problems, Taasoobshirazi and Glynn (2009) found that their ability to develop a successful strategy was influence by students' conceptualization of the problem and their self-efficacy, or belief in their own ability to succeed.
Walsh et al. (2007) carried out a phenomenographical study of problem solving approaches used by physics undergraduates. They found that four different approaches were observed: a scientific approach, which includes qualitative analysis of the situation, planning and execution of a solution, and evaluation of the solution; a structured plug-and-chug approach, which includes qualitative analysis based on formulas; unstructured plug and chug, which has a focus on required variable and lacks evaluation; a memory-based approach which uses analysis of previous examples and lacks evaluation; and no clear approach. The students who adopted a scientific approach were able to adopt the plug-and-chug approach when tackling less complex problems but plug-and-chug students were unable to adopt the scientific approach.
Thus, much of the published evidence in the area of problem solving in chemistry has focussed how students tackle algorithmic or structured problems or on what factors affect their performance in more open-ended problems. Our focus in this study is on the approaches that students adopt when faced with open-ended or complex problems in chemistry.
The think aloud method was chosen for this study to enable analysis of the process of solving problems, rather than focussing on the problem solutions or scores, that is, approach rather than ability. The study took place through one to one sessions with the researcher. Students were encouraged to verbalise their thoughts and to write as much as possible whilst solving the problems. Intervention from the researcher was kept to a minimum and care was taken not to provide help with the problems themselves. The sessions were audio recorded and all problem solving scripts retained for later analysis.
The problems used all used a real-life context, did not provide the students with all the data required to solve them, and did not lead to a single correct answer. Although the chemistry content would be familiar to most students entering higher education, it became clear during the sessions that the style of the problem was unfamiliar to them. Students were not allowed to use calculators or to look up any information that they thought they required. This constraint was applied in order to study whether students were able to make sensible estimations and carry out approximate or order of magnitude calculations. Therefore, in order to solve the problems students would have produce a strategy, make sensible estimations and approximations and carry out simple ‘back of the envelope’ calculations and think qualitatively about the problem before generating any data to enable application of any algorithmic operations.
Johnstone (1993) has previously categorised problems according to whether the strategy required is familiar to the students, whether they are provided with all required data, or whether there is a clear goal (Table 1). Algorithmic problems are defined as type 1, as all required data is given, the method is well known and there is a single correct answer. At the other extreme of this categorisation are type 8 problems, in which data are incomplete, the strategy is unfamiliar and has to be developed, and the goal is ill-defined or there is no single correct answer. The problems used in this study are described as type 8 as, even though there may be some element of algorithmic calculations involved, students cannot use a well know routine and have to develop their own strategy, the data is not all given in the problem and they have to make estimations in order to develop their own data, and there is not a single correct answer to any of the problems. According to Funke (2010) these would be classed as complex problems.
Type | Data | Methods | Outcomes | Skills |
---|---|---|---|---|
1 | Given | Familiar | Given | Recall of algorithms |
2 | Given | Unfamiliar | Given | Looking for parallels to known methods |
3 | Incomplete | Familiar | Given | Analysis of problem to decide what further data are required |
4 | Incomplete | Unfamiliar | Given | Weighing up possible methods and deciding on data required. |
5 | Given | Familiar | Open | Decision making about appropriate goals. Exploration of knowledge networks. |
6 | Given | Unfamiliar | Open | Decisions about goals and choices of appropriate methods. Exploration of knowledge and technique networks. |
7 | Incomplete | Familiar | Open | Once goals have been specified, the data are seen to be incomplete. |
8 | Incomplete | Unfamiliar | Open | Suggestion of goals, methods, consequent need for additional data. All of the above skills. |
The problems used have been used in previous studies (Overton and Potter, 2011) and are given below.
Problem 1: You've been on ‘Who Wants To Be a Millionaire’ and won £64000. You decide to treat yourself and some friends to a holiday in America. The flight from Heathrow to New York is 7 hours. To provide breathable air on an aircraft recirculation cells containing KO2 are used. Potassium dioxide reacts with the exhaled carbon dioxide as follows:
4KO2 + 2CO2 → K2CO3 + 3O2 |
K2CO3 + CO2 + H2O → 2KHCO3 |
What mass of KO2 would be needed on a Boeing 747 for this flight?
Problem 2: On November 13 2005 an explosion at a chemical plant in Jilin City in north eastern China released 100 tonnes of benzene and nitrobenzene into the Songhua River. Two weeks later, nearly 700 miles downstream, the spill flowed into the Amur river flowing through the Russian city of Khabarovsk. The Chinese and Russian authorities used activated carbon in water treatment facilities to stop the contaminants getting into the municipal water supplies. Quantities of carbon were also dumped in the river. Below is a table giving the specifications of activated carbon typically used for water treatment.
Grade | Filtracarb FY5 | Filtracarb CC65/1240 |
Type | Granular | Granular |
Surface area BETN2 (m2 g−1) | 1150 | 1050 |
CTC (%) | 55 | 65 |
Bulk density (g cm−³) | 0.49 | 0.45 |
Hardness (%) | 99 | 90 |
Iodine no. (mg g−1) | 1100 | 1050 |
What mass of activated carbon would be required to completely eliminate the pollutants from the water in the affected areas?
Problem 3: Many commercial hair-restorers claim to stimulate hair growth. If human hair is composed mainly of the protein α-keratin, estimate the rate of incorporation of amino acid units per follicle per second.
The student participants were volunteers drawn from years 1, 2 and 3 of full-time undergraduate chemistry programmes. They covered a range of abilities, gender and ethnic backgrounds. The sessions were not time limited but all students solved three problems within one hour. The interviews were conducted over two academic years, with 15 students involved in year 1 and 12 in year 2. The study in year 1 led to a limited number of approaches being identified so the second study was carried in order to verify those results.
The students' scripts were analyzed along with the audio recordings of the interviews by each researcher in order to identify the approaches used as each student solved each problem. Qualitatively distinct approaches were identified for each solution and each recording. Several different approaches could be identified for each student as each individual segment of activity was analysed separately. This process was carried out independently by each researcher (Armstrong et al., 1997) and a high degree of consensus was found between the themes and approaches identified by each, thus providing confidence in the categorisation. The individual interpretations of the protocols was compared and discussed and a common set of observational categories emerged from that process. This initial set of categories was refined by collapsing together categories of a similar nature.
Each identified category was given an appropriate code to allow further analysis of each participant's data. For categories that described an ability and its corresponding inability, the code was given the notation of a single letter and a ‘+’ or ‘−’ sign, any other codes were given a two letter notation. For example, the ability to make estimations and approximations was given the code E+ whereas an inability to make estimations and approximations was given the code E−. The categories identified from the analysis of the students scripts and recordings are shown in Table 2. Once the initial categories had been identified and coded then the individual student transcripts and recordings were analysed and these codes were applied.
Category | Code |
---|---|
Makes estimations, approximations, generates data | E+ |
Unable to make estimations, approximations or generate data | E− |
Understands problem, what needs to know or do | P+ |
Can't get started, can't identify what needs to know or do | P− |
Logical approach and reasoning | L+ |
Not logical, gaps in reasoning | L− |
Makes sensible assumptions | A+ |
Guesses | A− |
Evaluates answer, aware of limitations | EV |
Seeks algorithmic approach | AL |
Distracted by context of problem | CO |
Lack of knowledge a barrier | KN |
![]() |
A shaded cell in the table indicates that this approach was observed for that participant whilst solving any one of the problems. The supportive, or positive, approaches have been shaded in light grey, unsupportive, or negative, approaches in dark grey. Table 4 presents the results for the individual students re-ordered in such a way that students who exhibited predominantly negative approaches and those who exhibited predominantly positive approaches are grouped together. Those that remain used a variety of approaches across the three problems. On viewing this presentation of the data it became clear that the students could be organised into three broad groupings based on the combination of approaches they used; those that use predominantly positive approaches like students 1, 11, 12, and 13, those that use predominantly negative approaches like students 6, 8, 10 and 15, and those that use a mixture of the all approaches depending upon the problem, such as students 2, 3, 5 and 7.
![]() |
The students who adopted mostly negative approaches were typically unable to identify what they problem was asking them to do and were unable to make estimations or to generate their own data. This approach is typified by student 11 in the extract from their interview:
Student: One follicle per second…I can't even answer that, I don't know where to start. I don't how long a protein takes…I don't even understand it
Researcher: Well what do you think it means?
Student: I know you want the number of amino acid units going into a follicle; an answer in unit per second. From this….no idea how to get those numbers. I have no idea where to get any numbers from.
Student 12 also demonstrates negative approaches and guesses at an answer:
Student: I don't understand the question.
Researcher: What do you think it means?
Student: I think it's about the rate of hair growing. It must be rate…, nothing comes to mind.
Researcher: What do you think you need to know?
Student: The speed that hair grows…or I think how many amino acids get into hair.
Researcher: That's what you have to work out.
Student: That's a funny question. Something like 5 or 6 amino acids a second…yeah, that's my answer.
Students who adopted mostly positive approaches could typically clarify the aims of the problem and were comfortable identifying what data were needed and making estimations. They were also likely to explore different strategies and evaluate their answers. This is illustrated by this extract form the interview from student 6:
Student: The basic knowledge I have is 12 amino acids make up polypeptide. So I need to know the rate of amino acid production I suppose …have to be average wouldn't it? Unless I work backwards from a bald patch and work out how fast hair grew, how long from start to finish and work backwards. I'd need a designated site to measure and need to know how many follicles in that. or I could do it on how fast hair grows which is directly proportional to the rate of amino acid production. Yeah, that's how I'd go about it. Go for one hair. How fast does that grow? A length, a time and composition. I say 2 days to grow 2 mm and need certain number of amino acids a to produce that. So.
Researcher: So what are you thinking now?
Student: I'm thinking about estimating the volume of the hair and an amino acid and then I can use that with the volume of hair. So if I say an amino acid is diameter 10 nm and that hair is 0.1 mm thick. So I can find how many can fit in that volume of hair.
So, I've worked out the volume of an amino acid molecule and divided volume of hair which gives me this answers. That seems a lot to me. I'll do that again.
Student 8 also demonstrated these positive attitudes and developed an effective strategy very quickly and was very comfortable making estimations.
Student: So I guess hair grows around 1cm a month…so that's 0.03 cm a day…and 3.85 × 10−7cm a second. So if hair has a diameter of, say, 0.1 mm that's 0,01 cm..and I can find the volume of hair per second.
Researcher: What are you doing now?
Student: I'm mapping out spherical amino acid units. If I know the width of a unit I can work out the volume and then can divide into the volume of hair to find the answer. An amino acid is, let's say, 10 nm, that's 10−8m, ah, 10−6m, no it must be smaller, it's 10−10cm…
Students who adopted a mixture of approaches demonstrated behaviours typical of both these situations described above and their approach varied with the problem. They seem to be more influenced by the context of the problems, as illustrated by this extract from the interview of student 7:
Student: Follicle per second? I need the formula of keratin. It says it in the question. So do you want a rate equation? Or a number?
Researcher: Well, how would you work that out?
Student: I need to know what k looks like. Is it an amino acid? But just trying to work out the way to go.
Researcher: What have you written there?
Student: A rate equation and so I want the concentration of the product… So suppose I could guess how much amino acid there is, how much hair restorer you could put on your head at one time…guestimate how much keratin…could shave my head and work it out. What is concentration of keratin in my hair? Half a kilo? A follicle? How do you think of these questions? Well, it's supposed to stay in your hair and make it grow not to come out, so it must be a low rate.
Student 5 typified how some students became distracted by their own prior knowledge and the context of the problem.
Student: How do you monitor that? You'd have to take a hair out each second. The best way to monitor amino acids is by mass spec…keratin is a helix, a regular helix, it's hollow too, I've done this before…amino acid per follicle per second?…it will have to be a really rough estimate unless..this is awkward because if you pull out hairs they'll all be different lengths, they won't all grow at the same rate. How can we monitor hair growth? If you get someone with hair and cut it off to the skin and then let it grow and, measure the rate of growth…then pluck out one hair and analyse it by mass spec because it's really sensitive.
The focus of this study was an exploration of the approaches students used when tackling problems of this type rather than problem solving ability. Therefore, the problem solutions were not formally marked. However, it was clear that only those students exhibiting predominantly positive approaches produced satisfactory solutions to one or more of the problems. Those using negative approaches rarely made progress towards a solution and the performance of those using a mixture of approaches varied from student to student and with the individual problems.
Those students who adopt the positive approaches could be described as taking a scientific approach; they can understand a problem, are logical, make estimations, can handle a lack of data and evaluate their solution. This description corresponds to one of the types of problem solver described in Walsh's (2007) study of solving algorithmic problems in physics.
Those students that adopt negative or unhelpful approaches to solving open-ended problems take an unscientific approach and are unable to define for themselves what the problem is about and find dealing with a lack of data or a lack of relevant background knowledge particularly difficult to cope with. These students never evaluate their solution are often distracted by the context of the problem and seek an algorithmic solution. This class of students corresponds to Walsh's ‘no clear approach' in the study of physics students.
Perhaps the most interesting group of students is the third group. These students use a wide range of approaches depending on the problem encountered, and often within a single problem. Whether they could identify the problem and handle lacking data seemed to depend on the individual context. These students usually evaluated their solutions but they also sought algorithmic approaches. That students seek algorithmic routes through these problems is not surprising when most of their previous experience in problem solving has been of the algorithmic, structured type (Bennett, 2008; Pappa and Tsaparlis, 2011).
Bodner and Domin (1991) and Cartrette and Bodner (2010) describes students who take an unstructured approach to problem solving as novices and those that take a structured or scientific approach, as experts. It would seem reasonable to assume that the students in this study who exhibited all positive approaches are indeed functioning as expert problem solvers. Those who adopted only unhelpful approaches are clearly novice problem solvers as they find solving these complex problems challenging and have little success. Those who use the wide variety of approaches depending on context may be in a transition phase from being novice to expert problem solvers. The three types of problem solvers did not correlate with age or year of study, so this transition, if indeed it is a transition, must be related to intellectual development, practice or cognitive factors.
There are, of course, limitations to this study. This was a small investigation of 27 students solving just three problems. The sample is size is small from which to generalise and the individual problems selected could have influenced the outcome of the study. As with any qualitative study, the role of the investigator can influence the data collected and all interpretive methodologies are subjective to some extent. The sample will be extended in order to further validate these findings. The issue of whether our three types of students constitute novice, expert and transitional problem solvers will be investigated by identification of the characteristics of experts and novices. Longitudinal studies of the development of approaches to solving complex, open-ended problems will be carried out in order to ascertain whether these approaches are fixed for individuals or whether, with intervention, they can be accelerated to the expert/scientific stage.
This journal is © The Royal Society of Chemistry 2013 |