Moritz
Krause
a,
Stephan
Kienast
b,
Torsten
Witteck
c and
Ingo
Eilks
*a
aUniversity of Bremen, Biology/Chemistry, Bremen, Germany. Tel: +49 42121863280E-mail: ingo.eilks@uni-bremen.de
bUrsulinengymnasium Werl, Werl, Germany
cWestfalen-Kolleg Bielefeld, Bielefeld, Germany
First published on 27th June 2013
This paper describes the development and assessment of a learning environment to support the transition of students from lower to upper secondary chemistry education in Germany. The learning environment centers on reviewing and expanding learners’ prior chemistry knowledge in heterogeneous learning groups upon entering upper secondary education. The different learning materials are integrated in a computer-based learning environment, which offers a set of different learning stages which sequentially build upon each other. Another central element of the learning environment is an integrated test platform, which provides assessment and instant feedback to the students after completing each of the learning stages. This paper reports on the development of the learning environment by a group of teachers following the model of Participatory Action Research. Four schools and 81 students were involved in the implementation of the learning environment. Data sources used to reflect the learning environment were classroom observations, analysis of teacher feedback during monthly discussions, and a student survey. The study describes potential uses and reflects the benefits of using computer-based assessment within this learning environment from the viewpoint of the teachers and the students.
This paper reports on the development of a learning environment structured to help balance out differing levels of prior knowledge upon entering upper secondary chemistry education and to prepare the students for the demands of the forthcoming chemistry lessons. The environment offers a set of different learning stages which sequentially build upon each other like in a bicycle race. This is why the lesson plan was called the Tour de Chemie (‘Chemie’ is the German word for Chemistry). The learning materials for the different stages were integrated in a computer-based learning environment to allow for flexible use and autonomous learning. The goal of the learning environment is to provide the learners with a framework allowing them to individually and cooperatively review prior learning and expand their knowledge. Learning with the Tour de Chemie should take place in a self-directed and progressively structured atmosphere. Nevertheless the students should continuously receive feedback on their learning progress. Feedback and assessment are both important for the restructuring of the learning environment by the teacher, as well as for supporting the learning progress of the learner (Regan et al., 2011). Thus a test platform has been designed, which provides computer-assisted assessment of the student learning progress after each learning stage, and forms an integral part of the learning environment.
This paper discusses the development of the learning environment by a group of teachers following the model of Participatory Action Research in chemistry education (Eilks and Ralle, 2002). It describes the developed learning environment and reflects its application from the viewpoint of the teachers, external researchers, and the students. In addition to the presentation of the teaching approach the study intends also to contribute an answer the question: how beneficial is it to use computer-based feedback systems to support more thoroughly individualized, autonomous and cooperative learning of chemistry in heterogeneous learning groups?
Bell (2007) suggests that assessment should move away from purely traditional psychometric testing towards educational assessment to improve learning. Assessment should be interwoven with the process of instruction and should become an integral part of the teaching and learning process. Shwartz et al. (2013) also suggest that assessment is important for learning and that a combination of formative and summative assessment should be applied, which continuously accompanies the learning process. Black and William (1998) have documented that strengthening formative assessment can increase learning gains.
Educational theory suggests the use of a wide variety of assessment techniques, involving paper and pencil tests, concept mapping, interviews, portfolios, etc. (Shwartz et al., 2013). As long as the teaching and learning scenario does not mainly focus on higher-order learning skills, traditional forms of assessment like multiple choice or open-ended questions are still an appropriate component in the portfolio of assessment techniques (Black, 1998). However, the entire curriculum still requires a good mix of different forms of assessment (Shwartz et al., 2013). Despite the call to apply a wider variety of assessment techniques, assessing subject matter knowledge with the aid of pencil tests with multiple choice or multiple select items will still maintain their place in the assessment practices (Tamir, 1998; Shwartz et al., 2013).
Traditional forms of assessment like multiple choice, multiple select or content-focused questions in an open answer format can be easily applied in paper and pencil tests. However, thanks to the progress of modern computer technology it has become quite simple to transfer such question types to computer-based assessment forms and online platforms without any significant differences in effectiveness (Lee and Weerakoon, 2001). The general benefits of computer-based assessment platforms have already been discussed by Bull and McKenna (2004), Charman (1999) and Walker (2010). Taken together with specific examples from chemistry education, e.g.Cole and Todd (2003), David (1992), Lowry (2005), Bertolo and Lambert (2007), Ryan and Dunne (2011) and Ryan (2013), computer-based systems’ specific value can be seen in their efficiency while automatically marking responses, in providing immediate learner feedback on any potential weaknesses, and in being available upon demand and, in the case of larger learning groups, simultaneously accessible by all pupils.
Assessment is of specific importance for heterogeneous learning groups. It is the key which allows the teacher to adapt the teaching and learning environment in the sense of differentiated teaching and learning (Black, 1998). Also, individual feedback is necessary for increased levels of autonomous and self-directed learning. Providing each student with regular and instant face-to-face feedback after different learning stages is highly demanding, intensely time consuming and barely manageable in large learning groups (Lowry, 2005). By 1976 Dunkleberger and Smith had already suggested taking advantage of developments in computer technology to effectively deal with individualized feedback in heterogeneous chemistry classes. A computer-based assessment tool might help to partially – or possibly even completely – ease the burden placed upon the teacher when preparing, carrying out and evaluating a wide variety of individually adopted student tests (Walker, 2010).
Walker et al. (2008) describe the existing studies on students’ experiences with computer-based or online assessment as being limited in scope. This seems to be especially true in the case of secondary school chemistry education. It seems that computer-based and online assessment and feedback is more thoroughly applied at the tertiary level, where some evidence is available. A quite good summary and report of this situation was provided by Hepplestone et al. (2009). More specifically, Walker (2010), for example, described findings that undergraduate students consider online assessment as a suitable tool in science education. Both Walker (2010) and Rickerts and Wilks (2002) found that students perceive online assessment as less stressful than conventional tests. They do not feel worry or concern when working with tests operated by computers or online testing platforms. Kennepohl et al. (2010) have documented high levels of predictive power for online self-diagnostic assessment with regard to later success in examinations. Ryan (2013) even suggested the development of self-assessment multiple choice questions for the promotion of chemistry learning. Finally, Cole and Todd (2003) and Lowry (2005) both documented that computer assisted instruction and formative assessment in undergraduate chemistry education can help to promote students' performance in summative assessment.
In this particular project, one PAR group took the initiative. This group has been working on different projects of chemistry curriculum development for 15 years (Eilks and Markic, 2011; Mamlok-Naaman and Eilks, 2012). The development of the learning environment presented in this paper took place over a time of roughly one year, with meetings of the PAR group occurring once a month. The group was composed of ten secondary school teachers from different schools, accompanied by a chemistry educator from the University of Bremen, Germany. The members of the group had widely varying professional experience ranging from about fifteen years to over thirty years of teaching experience.
Testing of the learning environment has so far been applied in five learning groups from four different grammar schools. The application included aspects addressing the equalization of heterogeneity present upon entering upper secondary chemistry education and creating a review and training module in advance of the final examinations taking place at the end of upper secondary education. Eighty-one students participated in the testing rounds. Data were collected through classroom observations made by an external researcher, joint analysis of teacher feedback within discussions of the PAR group during the monthly meetings based on minutes taken during these meetings and a written teacher feedback questionnaire, and carrying out a student questionnaire combining open and Likert-type questions (Table 1). All data were qualitatively analyzed. Because of the small sample and the potentially different points of view of teachers, students and external researchers coherence was checked by triangulating the findings from the different sorts of data and the different perspectives of the persons involved. This triangulation took place in a cooperative process with the teachers and accompanying researchers and the results were finally discussed and reflected by the whole PAR group. During the first rounds of testing also the student responses to the tasks were monitored to check whether all items could be solved by at least some of the students and that there was a good mixture of easy, medium and more difficult tasks.
Teacher | PAR group discussion minutes |
Open feedback questionnaire | |
Student | Open and Likert-type feedback questionnaire |
External researcher | Classroom observation protocols |
Particles, atoms and atomic mass | ˙ Matter is composed of particles. |
˙ Dimension of particle structures by working on examples from scanning tunneling microscopy. | |
˙ Sizes of the atoms and the atomic mass unit u. | |
Molecules and molecular mass | ˙ Division of pure substances, compounds, elements and mixtures. |
˙ Atomic structure, covalent bonding and ionic bonding. | |
˙ Molecules and molecular masses. | |
˙ The Law of the Conservation of Mass | |
The mol | ˙ Relationship of the macroscopic mass unit “g” and the submicroscopic mass unit u |
˙ Avogadro number | |
˙ The mol and its use | |
Molar mass | ˙ Molar masses |
˙ Examples for calculating their values | |
Molar gas volume | ˙ Behavior of ideal gases |
˙ The molar gas volume at standard conditions | |
Concentration | ˙ Concentration |
˙ Percent concentration and molar concentration |
Each stage is based on two pages of teaching material and respective computer-based assessments. The material pages review the content and are designed to be learned by the students individually or in small groups. Simple repetition and easy transfer tasks are given in accordance with the overall aim of the idea of the Tour de Chemie. Additionally, either the textbook or searches for information on the Internet (based on a list of pre-selected Internet pages with reliable information) can be used by the students. The students are free to decide how the cooperative groups of learners will be formed, strategies for dealing with the teaching materials, and the overall amount of time they want to spend on individual materials and tasks. Nevertheless, a joint strategy for the learning group is also recommended. The students are asked to start with the first learning stage and become familiar with the materials during the course of their learning journey. Three steps are recommended to the students at the beginning:
˙ Each pupil works alone at reading the texts, then marks or writes down the most important passages or content which he or she does not understand.
˙ Then the learners reproduce the content of the text in conversations with each other and discuss the hard-to-understand sections. In case more information sources are necessary, they can be selected and used at the students’ discretion.
˙ The most important aspects are then noted in the pupil's individual workbooks and mastered.
As soon as students feel that they have mastered the content in the respective stage, they inform the teacher and sign up for the exam for that stage. These exams are used as a check of the material learned. The assessment of the learning is done by a computer-based tool developed using the software Question Writer leading to an HTML-based testing platform compatible with commonly-available Internet browsers (see www.tour-de-chemie.de.vu for German examples). The first assessment at each stage consists of 12 questions. The testing platform randomly selects these questions from a pool of possible questions at each learning stage. For the first assessment, single choice questions are applied. Single choice questions are questions in a multiple choice format with only one right answer to be selected. In this case each question had three to four wrong optional answers as distractors. Distracting choices were constructed by considering typical learning difficulties and misconceptions as described e.g. in Taber (2002), Kind (2004) and many research articles, and as suggested by Cole and Todd (2003) for respective assessment platforms. Distractors were based on any existing information available reported in research results as well as from the teachers’ experiences. Pupils must achieve a minimum score of 80%, which means a maximum of two mistakes are allowed. If successful, learners enter the next stage and the process begins again at the next higher level. If the students are unsuccessful, they receive their score and the computer-based platform provides step-by-step feedback for each task they completed.
The average testing time is less than five minutes, so that the overall time spent on the assessment platform is not extensive. If access to computers is limited, the learning materials can also be printed out, with only the assessment done on the computer. In the latter case, for a class with roughly 20 pupils about six computers are enough, since the students work with different strategies, at various speeds, and thus access the computers at different times. For fostering cooperative learning among the groups, a restricted number of computers is even an advantage. It is expected that the students working with paper printouts will more thoroughly concentrate on joint learning and the task of discussing any potential knowledge gaps. The cooperative learning atmosphere can also be fostered through the spatial separation of working places for contemplation of the texts stemming from the computer assessment stations. This aids in avoiding distraction of the students by the presence of digital media during the preparation phases and in sidestepping distraction in the testing phases arising from background noise during the group discussions and clarification exercises.
Two or more mistakes in the first assessment tool means that the stage has not yet been completed. The students are not allowed to move to the next stage (although there is no technical restriction). The students are asked to return to their learning materials again and master them. Students who fail at the same stage are asked to form groups and search for common misunderstandings in order to overcome them. Working in new groups with other partners or with additional material of their choice allows the students to find a different approach towards the content matter. One aid for working on their mistakes and to look for additional materials is the feedback they receive from the assessment system itself. The computer tool not only offers an overall score, but also reports step-by-step which answers were (in)correct. For incorrect responses, the correct answer is provided. Finally, the learners are asked for a second assessment. The students are retested through a broader range of question types than those used in the first round. Multiple select, questions in a multiple choice format in which more than one right answer is given and have to be selected, and open-answer questions (with only one word or short phrase answers) are now employed by the computer to provide the student with a different form of assessment. The students have to answer six questions, five of which need to be answered correctly in order to pass on to the next stage.
In the case that a student fails both assessments in a given stage, final assessment is performed via a short oral exam with the teacher. This exam allows the teacher to analyze where any learning difficulties concerning the respective stage lie. The face-to-face situation also allows the teacher to seize upon the conversation as an indicator of exactly where any weaknesses can be found. This means that penetrating questions can be employed and that helpful hints and tips can be provided to ease both learning and understanding in the individual. Individual counseling becomes possible, because the majority of other activities taking place in the classroom are centered around learning and feedback embedded in cooperative, individualized and automated forms. The teacher therefore has the time and opportunity to help individual pupils who still have problems with single items or areas after two computer-based assessment sessions. The oral testing and consulting conversation should not take more than 5–10 minutes. Fig. 1 shows a schematic representation of the sequence of events in the Tour de Chemie.
![]() | ||
Fig. 1 Overview of the Tour de Chemie. |
The application of the Tour de Chemie described above is only one of its potential uses. By integrating all of the learning materials with the assessment tool in an online format it also became possible to use various elements of the learning environment in different ways. Single stages can be used at any point in the curriculum for single students whenever deficits appear in their knowledge base. Individual stages can be selected and used to bolster the learning efforts of specific pupils in a specific area. The materials can also be offered as a general review mechanism before an upcoming exam, since the testing platform offers each learner immediate and direct feedback for both their level of learning and the success of their learning efforts to date. Recently, the first stages of the Tour de Chemie were already applied in lower secondary school also after the respective topics had been taught for the first time in the chemistry curriculum.
The motivational effects of the Tour of Chemie were viewed in a positive light by both the participating teachers and the external researcher. The teachers stressed the highly-differentiating nature of the unit. They had the impression that their pupils were very motivated by the lesson plan and that the students were involved in extremely intense, subject-matter based discussions. Within the PAR group discussions, the teachers reported and uniformly agreed that, despite the potentially dry, boring nature of a review of pure theoretical content matter, the students reacted in a very emotional, personally engaged fashion to the lessons. The learners reacted very positively to successes or openly showed disappointment when they failed one of the various tests at the end of the various stages.
In the written teacher feedback questionnaire to be completed after the lesson plan three questions were asked. The questions concerned the benefits of the Tour de Chemie, chances for further improvements, and the teachers’ consideration of advantages from transferring the Tour de Chemie materials and assessments into a multimedia-based learning environment. The main benefit, as described in all the teachers’ questionnaires, was the high degree of students’ autonomous and self-directed learning, especially by their chance to select their own learning pace as: “The benefit lies in the chance for the students for self-directed learning. With respect to their own prior knowledge and learning progress they can decide when to approach the next stage. Slower student can stay longer on a given stage; faster students can move forward earlier. The students are able to define their own learning speed autonomously. A further advantage is that the assessment is not done by the teacher. This allows the students’ learning with less fear and in a more comfortable atmosphere.”
Concerning the pedagogy, there were a few critical comments that were also later discussed in the PAR group. These comments mainly concerned how to deal with students so far being inexperienced in self-directed and cooperative learning. Those students sometimes started trying to solve the Tour de Chemie as a computer game in a trial and error approach: “I see problems if students try to answer the tasks on the computer without having worked on the materials before. The students then get feedback that their prior knowledge is weaker than they have expected. Some of them then try to fill the gaps in their knowledge by looking at specific aspects in the material, but still do not start really comprehensively working on the materials.” The teachers agreed that for such cases the computer environment so far has not provided enough guidance. However, it was also discussed whether this needs to be added in the computer environment or whether taking care of this phenomenon might be better a task to be assigned to the teacher.
The benefit of having moved the learning materials into the computer environment was seen in the higher flexibility of the system, the chance to work on the materials and tasks outside the classroom via the Internet, and the direct feedback with the inclusion of the mistakes made without the teacher first correcting the written test. Due to the automatic correction of the test the teacher has more freedom to support individual students. Also the fact that the teacher does not see the tests was considered as a positive element since feedback in this case was thought as formative feedback and not summative grading.
From the classroom observations it quickly became clear that developed experience with cooperative learning methods helped the students to follow the lesson plan. Students with rich experience in cooperative learning from lower secondary science education repeatedly changed working partners or formed new groups during the unit. This was largely determined by the overall progress made by individual students and the phase in which they found themselves at any given moment. Without teacher intervention, different learners also asked for assistance from pupils who had already successfully completed the stage in which they were having difficulties, e.g. by addressing the right and wrong answers given in the feedback from the assessment platform. The same held true for the preparation period before an oral examination by the teacher. This allowed lower-achieving learners to receive help from better-performing classmates.
As well as among the teachers, the student reflections on the lessons most frequently mentioned the perception of autonomy during the learning and working periods, which were often positively viewed with respect to the activities taking place in small learning groups. Many of the pupils valued the fact that they had to initially work alone on the texts before entering into partner or group work in the ensuing discussion phase. One student wrote upon an open question about the overall consideration of the lesson plan: “The Tour de Chemie is structured for the individual and helps us to work alone, while simultaneously supporting one another. We could clarify questions among ourselves and strengthen our knowledge through the process of exchange.” Many pupils also stressed the ability to choose one's own learning pace, as it proved to be valuable to support learning in heterogeneous groups (Gable and Herron, 1977). Two students stated: “I really liked the parts where we could make our own time schedules for working on the entire topic” respectively “We were allowed to learn following our own pace and thus were able to learn better”. The aspects of individual and self-assessment during the learning progress and the possibility of exchanging ideas with other students were both frequently mentioned as positive parts of the process: “We were asked to learn everything on our own only supported by learning materials and we were able to directly assess on the computer whether we understood everything right”. The assessment was also considered to be motivating: “We were more ambitious to solve the tasks correctly. The 80%-threshold was like a gold medal in the Olympic Games. At all costs, every one of us wanted to pass the 80%. It was fun answering the questions with the computer”.
When the learners were asked for the biggest difference between traditional classes in chemistry and this unit, one student replied in the open questionnaire: “The biggest difference was that there was almost no frontal teaching involved. The material was entirely completed in a self-sufficient working mode. At the end of every phase we had the chance to test our own knowledge. Learning took place less individually and more in group-based situations, so that many questions could be immediately clarified in the group.” The students also came to the point of self-direction in organizing their work and selecting their learning speed: “I liked the lesson plan since we students were able to make our own time schedule for working through a whole module. By the fact that every one of us was able to determine her or his own speed in learning we all got the chance to define own points of special emphasis” or “we are able to spend as much as time at a single stage as we needed.”
The feedback found in the Likert-questionnaires also supported the positive opinions expressed in teacher feedback, classroom observations and the open student questionnaires. Fig. 2 presents the Likert-items and the student responses. Learner autonomy and self-directedness in learning were especially appreciated by the students. They also acknowledged intensity and engagement of working due to the chosen pedagogy and the systematically embedded assessment.
![]() | ||
Fig. 2 Students’ responses on the Likert-items (N = 81). |
Criticism was rare in both the open questions and teacher feedback. Only one learning group described the application of the Tour de Chemie in less than enthusiastic terms. All schools but one are located in environments where exchange between both schools and learning groups at the beginning of the upper secondary level is quite large. In one school from a rural area, however, this was not the case. In this particular school, nearly all students in the new course had attended the same lower secondary school, learned with the same curriculum and had the same teacher. The heterogeneity in this school was low and the average achievement level was high. These students mastered all stages more-or-less in parallel and in a much shorter amount of time than the other groups. Nevertheless, the lesson was valued by the class, which described it as a review and a chance to assess the basic knowledge which had been learned at the lower secondary level.
We acknowledge the support of the German Fund of the Chemical Industry (FCI) for the development of the learning environment presented here.
This journal is © The Royal Society of Chemistry 2013 |