M. T.
Segedinac
a,
S.
Horvat
b,
D. D.
Rodić
b,
T. N.
Rončević
b and
G.
Savić
*a
aUniversity of Novi Sad, Faculty of Technical Sciences, 21000 Novi Sad, Serbia. E-mail: savicg@uns.ac.rs
bUniversity of Novi Sad, Faculty of Sciences, 21000 Novi Sad, Serbia
First published on 28th March 2018
This paper proposes a novel application of knowledge space theory for identifying discrepancies between the knowledge structure that experts expect students to have and the real knowledge structure that students demonstrate on tests. The proposed approach combines two methods of constructing knowledge spaces. The expected knowledge space is constructed by analysing the problem-solving process, while the real knowledge space is identified by applying a data-analytic method. These two knowledge spaces are compared for graph difference and the discrepancies between the two are analysed. In this paper, the proposed approach is applied to the domain of stoichiometry. Although there was a decent agreement between expected and real knowledge spaces, a number of relations that were not present in the expected one appeared in the real knowledge space. The obtained results led to a general conclusion for teaching stoichiometry and pointed to some potential improvements in the existing methods for evaluating cognitive complexity.
Our research introduces a novel approach to the evaluation of experts’ comprehension of the way in which students understand the domain. The approach is based on knowledge space theory (KST), and it allows the construction of the knowledge structure that experts expect students to have and the real knowledge structure that students demonstrate on tests and the comparison of the two. The proposed approach is applied to the domain of stoichiometry, and from the analysis of the differences between the expected and real knowledge structures, general implications for teaching stoichiometry are derived.
If students’ knowledge is to be modelled using KST, it has to be represented by a finite set of problems, such that each of them has exactly one correct answer. That means that KST is suitable for representing conceptual knowledge, but it is not adequate, for example, when representing tacit knowledge. The set of problems – called a domain – should be large enough to contain fine-grained, representative coverage of the field (Falmagne and Doignon, 2011). In fact, each task of the domain has to clearly represent the skill required to address that task. It means that a particular student's skill could be represented by several tasks of various complexities.
Skills that a student possesses are being represented by a set of problems which could be answered (solved) correctly under ideal conditions, namely the knowledge state (Falmagne and Doignon, 2011). Taking into consideration that assessment never occurs under ideal conditions, students’ answers are always afflicted by lucky guesses and accidental mistakes. Therefore, the knowledge state cannot be observed directly, but it has to be inferred from the student's answers and the characteristics of the domain itself.
For a test with n problems, there can be 2n knowledge states. This means that the set of knowledge states for a non-trivial domain is very large; for example, if a test contains 20 problems, there will be 1048576 potential knowledge states. Luckily, most of these knowledge states cannot be achieved by a student. For instance, if a student is not able to solve a task wherein the mass of a product is required, and the mass of a reactant and the chemical equation are given, he or she will not be able to solve an analogous task wherein the equation is not given.
This fact allows us to construct a knowledge structure that represents the set of all possible knowledge states for a given domain. For every domain it is assumed that there is an empty knowledge state, demonstrating that a student has mastered none of the skills required to solve the problems and that there is a knowledge state containing all the problems, representing that a student has mastered all of the assessed skills.
One particularly significant family of knowledge structures is that in which mastering new skills represented by the domain problems always leads to some knowledge states in the domain. In such knowledge structures, for any knowledge states the K1 and K2 union (K1UK2) is a knowledge state as well, meaning that the knowledge structure is U-closed. Such knowledge structures are identified as knowledge spaces (Falmagne, 1989; Albert and Kaluscha, 1997).
An important characteristic of knowledge spaces is that they can be represented by a surmise relation that assigns a set of preconditions to each problem. For a set of problems, it is said that they are preconditions of a problem p if a student has to be able to solve them in order to be able to solve the problem p. The problem p surmises the problems in the mentioned set. A student needs the prior knowledge and understanding assessed by the problems surmised by the problem p for solving the problem p.
A special family of knowledge spaces consists of those in which each problem has exactly one set of preconditions, meaning that they are not only U-closed but also ∩-closed. In such knowledge spaces a surmise relation is quasi-ordering, i.e. it is reflexive and transitive, and can be defined as follows: “problem a surmises problem b, if we know that a student is able to solve the problem a, we can infer that student is also able to solve the problem b” (Segedinac et al., 2011). In this paper, students’ knowledge is represented by such knowledge spaces.
KST can give significant contributions to the chemical education due to the fact that knowing knowledge structures can help in overcoming discrepancies among teachers, textbooks and syllabi concerning conceptual organization and adequate instructional strategies. This research proposes one such method which compares the expected knowledge space, constructed by experts, with the real knowledge space, constructed by analysing test results.
(I) Those that construct a knowledge space by querying experts
(II) Those that are based on the analysis of the problem-solving process
(III) Data-analytic methods
When a knowledge space is to be identified by systematically querying experts, appropriate combinations of questions are selected using special purpose algorithms, and these combinations of questions are presented to the experts who assess if these combinations are feasible. From these data, the surmise relation is constructed on the set of problems. One such method uses the QUERY algorithm (Koppen, 1993). The main advantage of these methods lies in the fact that they allow construction of a knowledge space before evaluation of students’ achievement. The most serious flaw of these methods is the requirement to continuously involve the experts in the process of knowledge space construction (Segedinac et al., 2011).
Methods based on the analysis of the problem-solving process chunk the problems that are to be solved by students into sets of sub-problems. One such method that identifies a set of motives necessary to solve each problem is proposed by Schrepp et al. (1999). These methods, similarly to the previous family of methods, can be applied prior to the assessment of students’ knowledge. In addition to that, these methods can be formalized to some extent (provided that the experts just have to identify the set of motives for each problem) and, therefore, it is possible to reduce the experts’ involvement (Segedinac et al., 2011).
Data-analytic techniques identify surmise relations on the sets of items. These methods are based on the assumptions that there is a true surmise relation underlying the data and that the careless errors and lucky guesses make the data noisy. All of these methods basically share the same structure. Firstly, they construct a set of candidate surmise relations and, after that, the candidates are evaluated against the data using a specific fit measure. Finally, the best-fitting candidate is selected as the surmise relation (Ünlü et al., 2013). The main advantage of these methods is that they do not require the involvement of experts, at all. The most important drawback of these methods results from the fact that it requires students’ knowledge to be evaluated before the knowledge space can be identified (Segedinac et al., 2011).
The most well-known data-analytic method is the inductive item tree analysis (IITA) – a method that identifies a surmise relation on a set of items. An overview of algorithms based on IITA and their application for the organization of educational objectives is given by Segedinac et al. (2011).
Besides the construction of the quasi-orders (reflexive and transitive binary relation), it is very important that the quasi-order, obtained by the data-analytic method, optimally fits the data. In IITA, the idea is to estimate the number of counterexamples for each quasi-order, and to find the quasi-order where there is a minimal discrepancy between the observed and expected numbers of counterexamples covering the entire competing quasiorders.
The inductive construction of the quasi-orders is stated as one of the main advantages of this algorithm (Schrepp, 1999, 2003). However, the inductive construction can be criticized as it is possible that two implications together could cause intransitivity, added separately. This drawback is overcome by corrected and minimized IITA, the method which is utilized in this paper.
Regarding science education, one of the first applications of KST was related to the evaluation of an instructional strategy for teaching the concepts of pressure, density and the conservation of matter (Taagepera et al., 1997). In that paper, KST is used to identify the most probable learning pathways and to evaluate the efficiency of the applied instructional strategies.
Further research included various chemistry concepts such as organic chemistry concepts (Taagepera and Noori, 2000), bonding concepts (Taagepera et al., 2002), chemical and physical properties of matter (Tóth and Kiss, 2006), concepts of the plane of symmetry, chirality, isomerism (Taagepera et al., 2011), the concept of solutions (Segedinac et al., 2011), etc. It is worth mentioning that the KST framework has allowed researchers to identify the students’ critical learning pathways (e.g.Tóth and Kiss, 2006), as well as to compare them with experts’ critical learning pathways (Vaarik et al., 2008). Some novel applications of KST in science education include its combination with other methodologies, such as phenomenography (Tóth and Ludányi, 2007), with the aim of investigating students’ reasoning and changes in cognitive structures.
Literature review on KST in chemistry education indicated that stoichiometry has been one of the most extensively researched topics. Research has been conducted to determine learning pathways while solving stoichiometric problems (Arasasingham et al., 2004), to assess students’ understanding of stoichiometry (Arasasingham et al., 2005; Tóth, 2007), to compare knowledge spaces of the students who use different strategies for solving stoichiometric problems (Tóth and Sebestyén, 2009), and to determine the quality of students’ cognitive structure, i.e. to examine if it consists of interconnected or fragmented chemical concepts (Taagepera and Arasasingham, 2013).
In contrast to the examined papers, which mostly utilized KST either to evaluate instructional strategies or to track the development of scientific concepts, our research introduces a novel application of KST, namely, evaluation of experts’ comprehension of the way in which students understand the domain. The fact that experts observe the domain in light of their own knowledge influences their expectations of the students’ knowledge space. The approach proposed in this paper, therefore, includes the construction of the (i) expected knowledge space by experts and (ii) real knowledge space using a data-analytic method, comparing the obtained knowledge spaces, and analysing the differences between them.
In this research, we aimed to compare the knowledge space constructed by experts with the one identified by the analysis of students’ responses (using corrected and minimized IITA). Due to the fact that both methods result in ∩-closed knowledge spaces, it is possible to compare the obtained knowledge spaces by applying graph differences. The obtained differences were utilized to analyse experts’ understanding of students’ knowledge about the domain.
A surmise relation that determines a knowledge space can:
(I) be inherent to the logical structure of the domain itself, or
(II) can result from a pure contingency.
Sometimes experts overlook some of the surmise relations that occur in knowledge spaces constructed by analysing students’ responses. Such relations can be very valuable because they can help experts to refine their understanding of students’ learning.
Therefore, the main contribution of this paper is reflected in the fact that this application of KST allows the refinement of experts’ comprehension of students’ understanding of the domain.
(I) Constructing the test that is the domain for the knowledge spaces. In this particular research, the test has been taken from Horvat et al. (2016) with permission.
(II) Constructing the expected knowledge space for the given domain by analysing the problem-solving processes. This knowledge space is constructed by domain experts.
(III) Constructing the real knowledge space using a data-analytic method for the obtained students’ responses
(IV) Identifying the differences between the expected and real knowledge spaces
(V) Analysing the obtained differences
The testing was conducted on a sample of 82 students studying to be pharmaceutical technicians at secondary medical school “Dr Andra Jovanović” from Šabac, Republic of Serbia, 15–18 years old. In the chemistry curriculum of this educational profile, chemistry is realized through several obligatory subjects: chemistry, medical biochemistry, analytical chemistry and pharmaceutical chemistry. Stoichiometry is studied in the first year within the subject chemistry.
All the students were of a similar socio-economic status, predominantly the urban population. Testing was anonymous, and before testing, students were introduced to the goal and purpose of the research. All the participants, included in the study, accepted to willingly participate in the study without any constraint or expectation of reward. They were informed about the test two weeks in advance so that they could revise the topic on stoichiometry.
The test consisted of 20 stoichiometric problems covering the following chemistry concepts: writing and balancing chemical equations, stoichiometric calculations, and calculations with impure substances or solutions. Some of the problems required knowledge of additional concepts such as Archimedes' principle and the reactivity series of metals. The test with the cognitive complexity rating procedure is available in Appendix A (ESI†).
Cognitive complexity rating | Problems |
---|---|
1 | 1, 7, 14 |
2 | 2, 15 |
3 | 4, 16 |
4 | 3, 8, 9, 10, 18 |
5 | 11, 17 |
6 | 12, 19 |
7 | 5, 13 |
8 | 6, 20 |
The software tool (available at https://github.com/milansegedinac/kst_tools/tree/master/KSTFromRubricTool/KSTFromRubricTool with an example of its use shown in Fig. 1) that implements this above-mentioned procedure is developed as part of this research. The software tool takes cognitive complexity ratings as an input (file in the CSV format) and produces the knowledge space. The obtained knowledge space is given by its edges.
Fig. 1 The software tool for constructing the knowledge space from experts’ estimation of cognitive complexity. |
This cognitive complexity rating allowed us to construct the expected knowledge space, as shown in Fig. 2.
The given Hasse diagram (Fig. 2), which represents the expected knowledge space, shows, for instance, that problem 2 surmises problem 7, and that it is surmised by problem 4. This means that a student who is able to solve problem 4 can also solve problems 1, 7, 14, and 2. The most complex is problem 20, and, according to the expected knowledge space, if a student can solve this problem, he or she can solve any other problem on the test.
This approach is suitable when there are no experimental test results. One of the key characteristics of such an approach is that it reflects experts’ premises about students’ learning. Therefore, the comparison of such knowledge space and the one obtained by analysing the test results can point to some of the surmise relations that experts overlook, and therefore, it can help experts to refine their comprehension of students’ learning.
An estimation of a problem's complexity from a real knowledge space in this paper is based on the concept of the outer fringe of a knowledge state (Falmagne and Doignon, 2011). Within the context of this research, the outer fringe represents a set of items that a student is ready to learn (Falmagne and Doignon, 2011). Relying on the outer fringe concept, a procedure for calculation of cognitive complexity is defined as follows:
All items without predecessors have a cognitive complexity value k = 1.
All items belonging to an outer fringe set containing items with cognitive complexity k receive a cognitive complexity value k + 1.
For determining cognitive complexity from a real knowledge space, the software tool was developed which calculates this cognitive complexity by implanting the above-described algorithm. The application, which is available on the link https://github.com/milansegedinac/kst_tools/tree/master/KSTComplexityTool/KSTComplexityTool, takes a knowledge space (adjacency matrix) as an input and returns a list of pairs (itemNo, CogCompl), where ItemNo is the problem ID and CogCompl is the corresponding cognitive complexity (Fig. 4).
Table 2 depicts cognitive complexities calculated for the obtained real knowledge space.
Cognitive complexity rating | Problem |
---|---|
1 | 14 |
2 | 1, 7 |
3 | 2, 8, 9, 11, 16 |
4 | 3, 4, 17 |
5 | 10, 18 |
6 | 19 |
7 | 13 |
8 | 5 |
9 | 12 |
10 | 6 |
The knowledge space obtained from test results has a greater number of complexity levels than the one constructed by experts. Therefore, the scales that represent the cognitive complexity have a different number of ratings (in the expected knowledge space ratings range from 1 to 8, while in the real one they range from 1 to 10). This does not mean that a task that has a cognitive complexity rating of 6 in the real knowledge space has to be more complex than the one rated as 5 in the expected one. Thus, instead of comparing the numerical rates of cognitive complexity between the real and expected knowledge spaces, we propose to analyse the relative position of the problems in the two knowledge spaces and the relationships among the problems.
As part of this research, a software tool was developed to determine the difference between knowledge spaces. This software tool is available at link https://github.com/milansegedinac/kst_tools/tree/master/KSTComplexityTool/KSTComplexityTool, and an example of its usage is shown in Fig. 5. The tool takes two knowledge spaces represented as graphs (in this research these spaces were expected and real knowledge spaces) and produces a graph representing their difference as a set of edges (set of pairs of connected nodes).
There is a 71.32% match between the expected and real knowledge space graphs, where 71.32% of the expected knowledge space is represented in the real knowledge space. It should be noted that the real knowledge space contained a number of relations that were not present in the expected one. Those relations are shown as dashed arrows in Fig. 6. It should be noted that during the construction of the real knowledge space, problems 15 and 20 were identified as outliers, and, therefore, removed from any further observations. To facilitate the observation of differences, the annotated graph is available in Appendix A (ESI†).
The relations identified in the real knowledge space that were not present in the expected knowledge space are (14, 1), (14, 7), (16, 4), (8, 3), (8, 10), (9, 10), (9, 18), (19, 12), (13, 5).
The relations that are inversed in the real and expected knowledge spaces are (5, 12), (9, 4), (17, 10), (17, 18), (13, 12).
Furthermore, experts have rated problems 16 and 4 with a cognitive complexity value of 3 (Table 1), since both problems 16 and 4 include one easy and one medium concept. Both tasks include the chemical equation concept, and also calculating molar mass. However, what distinguishes them is the complexity of equations. Namely, the oxidation–reduction reaction equation has only 3 participants while a non-oxidation–reduction reaction equation has 5 participants. Constructing the table, the experts defined the determination of coefficients in the non-oxidation–reduction reaction equation as an easy concept, while the determination of the coefficients in the oxidation–reduction reaction equation was defined as the concept of medium difficulty. In doing so, they overlooked the fact that students can effectively determine coefficients in the simple oxidation-reducing equation using an inspection method, while they may have difficulty in using the same method to equate a complex equation that is a non-oxidation–reduction reaction. Further, if the inspection method is used, the procedure for equalizing chemical equations becomes more complicated if there are more sources of an element in equation (Staver and Jacks, 1988). If we compare the equations of the two above-mentioned tasks, it can be noted that in task 16 there is one source of K, one source of Cl and one source of O on both the left and right sides of the equation, while there are two sources of O on the right side of the equation of task 4, which makes it more complex. This example also showed that knowledge space theory recognizes nuances in complexity that experts failed to observe.
For problems 8, 3, 9 and 10, which are all rated with a cognitive complexity rating of 4, there are three newly established relations (8, 3), (8, 10) and (9, 10) in the real knowledge space. Problem 8 includes one difficult concept, while problem 3 includes two medium concepts and one additional concept according to the experts’ evaluation. This insensitivity of the rubric, which is reflected in the fact that different numbers and types of concepts are rated with the same cognitive complexity, led to the formation of a novel relation (8, 3) in the real knowledge space. If we add the fact that the equation in task 3 is quite complex, since on the right side of the equation we have two sources of N and two sources of O (as previously described), and the partial reduction of the nitrate ion, the cause of the inversion becomes clear. On the other hand, tasks 8, 9 and 10 include one difficult concept, and the newly established relation (8, 10) and (9, 10) could be explained by a different molar ratio between the given and required substances. In tasks 8 and 9, it is 1:1, while in task 10, it is 1:5. This is in accordance with the research of Anamuah-Mensah (1986), who found that problems involving 2:1 ratios caused various conceptual problems to students, which did not exist in problems involving 1:1 ratios.
A similar situation was observed in tasks 4 and 9. While the expected knowledge space placed problem 4 as a precondition for problem 9, in the real knowledge space it is the opposite way around, i.e. problem 9 is a precondition for problem 4. In both tasks, the given and required substances are different physical quantities; however, task 9 involves a 2:2 (i.e. 1:1) ratio, while task 4 involves a 2:1 ratio.
Problems 18 and 9 represent a typical example of two completely different tasks, which experts assessed with the same value of cognitive complexity, relying on the table. Task 9 contains one difficult concept while task 18 contains 3 easy concepts. However, although the concepts in task 18 are easy, working memory load should be considered. To successfully solve this task, the student has to recognize one of the two equations necessary to perform the calculation, then to know how to calculate the molar masses of the reactant and product, to deal with a complex chemical equation containing a large number of participants (which puts additional load on working memory).
Further, in the expected knowledge space, problem 10 is identified as a precondition for the problem 17, while in the real knowledge space, problem 17 is surmised by problem 10. This inversion results from the specificity of problem 17. If a student can write the stoichiometric equation in problem 17, he or she can read the amounts of a required substance directly, while problem 10 requires him or her to compare the number of molecules of the required substance with the mass of the given substance. A similar reason can explain the inversion of the surmise relation between problems 18 and 17. Instead of problem 18 being a precondition for problem 17, as expected by the knowledge space constructed by experts’ evaluation, it turns out that in the real knowledge space problem 18 surmises problem 17. This is probably because in problem 18 students have to compare the masses of the required and given substances, which requires a calculation of the relevant molecular masses, as previously explained.
There is also the new relation between problems 13 (2 easy and 1 difficult concepts) and 5 (1 easy and 2 difficult concepts), with a complexity rating of 7. This is an example of two completely different tasks of the same estimated cognitive complexity. A particularly important difference, which should be noted, is the fact that in task 5 it is necessary to know how to write a requested chemical equation. A similar situation is observed in tasks 12 and 13, as well as tasks 19 and 12. In the expected knowledge space, task 12 (2 difficult and 1 additional concept) precedes task 13 (2 easy and 1 difficult concept), while in the real knowledge space task 13 precedes task 12. Furthermore, problem 19 precedes problem 12 in the real knowledge space, which also requires writing a chemical equation. Numerous studies have shown that students often have no problem in equating chemical equations, commonly using the inspection method. However, it has been shown that students very often do not understand the essence of the underlying chemical change (Talanquer, 2011). Therefore, if a student does not understand the essence of a chemical change, he or she will not be able to present it with an equation and therefore will not be able to solve the task even if the request is simple.
Finally, the expected knowledge space assumes that problem 5 requires students to be able to solve problem 12, whilst in the real knowledge space it is the opposite. It could be explained by the fact that students have not mastered the reactivity series of metals. Namely, many of the students made a mistake, believing that the gas produced in the reaction of zinc and sulfuric acid is SO2 instead of H2.
The results of this study have shown that the real knowledge space is richer in levels of complexity than the expected one. This fact led us to compare the knowledge spaces themselves, and the comparison has shown that there is a finer structuring in the real one.
The proposed approach can help experts identify the fine structure of easy concepts. This structure is often neglected by experts, who tend to rate certain concepts as easy, finding them trivial, although those are not trivial to students who have not fully mastered them, potentially causing cognitive overload.
Some of the methods for sequencing learning experiences, e.g. cognitive complexity rating rubrics, place all the difficult concepts in the same category. The approach proposed in this paper can improve on this and help experts to identify a fine structure in the set of difficult concepts.
In addition to that, it can also point to some flaws in the existing methods for evaluating cognitive complexity. For instance, the method utilized in the paper by Horvat et al. (2016) rated the cognitive complexity same value to problems which include two difficult and one easy concepts, and those with one difficult and two easy concepts. The approach proposed in this paper suggests that there are cases in which this does not hold.
This approach is also important for the development of problem-solving strategies in stoichiometry and can help teachers in task design as it could point to some overlooked facts. In this regard, this research provides the following implications for teaching stoichiometry:
(I) Cognitive complexity is conditioned by the complexity of the reaction manifested in two ways. Firstly, students encountered more difficulties in equating chemical equations with a larger number of reaction participants. Secondly, the existence of more than one source of an element on the left or right side of the equation further increased the complexity of tasks.
(II) Students know how to equate the chemical equation using the method of inspection, without proper reasoning for underlying concepts.
(III) Tasks which include 1:1 ratios of given and required substances, or those that could be reduced to 1:1 ratios, are easier than those tasks which include ratios different from 1:1.
(IV) Incorrectly adopted concepts, which experts are sometimes not aware of, can cause an increase in cognitive complexity.
(V) Students preferably choose to compare amounts as direct stoichiometric coefficients rather than as masses where they have to calculate molar masses. It is important to note that this implication is strictly related to the students involved in this research, and could be a consequence of an applied instructional method used by teachers during regular classes or a wrongly acquired conception of molar mass.
Some of the recommendations for teachers would be to dedicate more teaching time to complex equations, especially those involving multiple sources of one element. It is especially important to reflect on the submicroscopic reality represented by a chemical equation in order to encourage meaningful understanding of chemical equation concepts. In addition, more examples of tasks with ratios different from 1:1 should be given. Finally, it is especially important to determine whether students already have some misconceptions.
The most important constraint of the proposed application of knowledge space theory is the fact that it requires experts to give a detailed analysis of the difference between the expected and real knowledge spaces. Each relation given by the graph deference has to be appraised by a domain expert. Our further research will, therefore, be focused on designing software tools that will assist and help experts in this task.
For the purpose of the construction of the expected knowledge space, a software tool was developed as a part of this research. The real knowledge space analyzed in this research was constructed using the DAKS library for programming language R. To identify the difference between the knowledge spaces, another software tool was developed. One of the paths of further research will be to develop a single software application that will put together the functionalities of constructing, comparing and analysing knowledge spaces that will simplify the use of knowledge space theory in educational research.
Finally, different instructional strategies will result in different knowledge spaces. This feature of the proposed approach can actually be utilized to compare different instructional strategies, which will be another path for our further research.
Footnote |
† Electronic supplementary information (ESI) available. See DOI: 10.1039/c8rp00052b |
This journal is © The Royal Society of Chemistry 2018 |