View PDF VersionPrevious ArticleNext Article
 
DOI: 10.1039/C4RP00009A (Paper) Chem. Educ. Res. Pract., 2014, 15, 628-638

How guidance affects student engagement with an interactive simulation†

Julia M. Chamberlain *a, Kelly Lancaster a, Robert Parson b and Katherine K. Perkins a
aDepartment of Physics, University of Colorado Boulder, Boulder, Colorado 80309, USA. E-mail: Julia.M.Chamberlain@colorado.edu
bDepartment of Chemistry and Biochemistry and JILA, University of Colorado Boulder, Boulder, Colorado 80309, USA

Received 6th January 2014 , Accepted 17th June 2014

First published on 18th June 2014

Abstract

We studied how students engaged with an interactive simulation in a classroom setting and how that engagement was affected by the design of a guiding activity. Students (n = 210) completed a written activity using an interactive simulation in second semester undergraduate general chemistry recitations. The same simulation – PhET Interactive Simulations' Acid–Base Solutions – was used with three written activities, designated as Heavy Guidance (HG), Moderate Guidance (MG), or Light Guidance (LG). We collected mouse click data and classroom field notes to assess student engagement with each type of activity. Simulation features were characterized as “prompted” or “exploratory” based on the presence or absence of explicit guidance in the written activity to use that feature. While students in every condition were engaged with the simulation and their activity, student interaction with “exploratory” features decreased significantly when more guidance was provided (LG = 85%, MG = 68%, HG = 9%, p < 0.0005). Lighter guidance groups explored more and attended to their simulation interactions, indicated by a redraw task in the week after use. These results indicate that activity design – in terms of guidance level – can strongly influence student exploration with an interactive simulation. We discuss the implications of these results for the design of activities to accompany simulations, including how to increase student practice in scientific inquiry.

Introduction

As simulations and animations are increasingly used for teaching and learning chemistry (Suits and Sanger, 2013), it is important to understand both the learning outcomes that can be achieved with these tools, and the ways in which students engage with them in educational settings. Simulations provide a scaffolded interface for learning through exploration (Lee et al., 2006; Adams et al., 2008a, 2008b; Plass et al., 2009; Podolefsky et al., 2010; Chang and Linn, 2013), and have been used with inquiry-based teaching methods to enhance science learning (de Jong and van Joolingen, 1998; Robinson, 2000; Rutten et al., 2012; Moore et al., 2013). When used in class, simulations become part of an integrated system of resources that guide learning – including implicit guidance from the simulation, as well as explicit guidance from the written activity and the instructor (Hennessy et al., 2006; Lee et al., 2006; Chang et al., 2008; Khan, 2011). While much research has contributed to present knowledge of optimal design elements for interactive simulations (Lee et al., 2006; Adams et al., 2008b; Plass et al., 2009), less is known about how the amount, timing, and type of guidance affects students’ interaction, interpretation, and learning with these educational tools.

Not surprisingly, the amount and type of guidance that students receive when they use simulations and animations can affect student exploration, perceptions of the relative importance of learning materials, and learning outcomes. Chang et al. (2008) found that using simulations to assist learners with hypothesis generation benefitted their learning, but providing directions for experimental procedures significantly limited students' exploration and impaired learning results. In a study comparing high and low levels of guidance in a written activity with a chemistry simulation, Akaygun and Jones (2014) found that less guidance promoted student focus on lesson content, whereas more guidance resulted in student focus on the structure of the activity, such as the number of questions and time required to answer them; learning gains were equivalent between these two groups. Gonzáles-Cruz et al. (2003) examined the effects of explicit guidance in different written activities using an enzyme kinetics simulation. Their comparison of heavy, moderate, and minimal explicit guidance during simulation use to a control (no simulation use), revealed that simulation use improved student learning overall, and that different levels of guidance benefitted students at different times. The moderate guidance group performed better in the short-term on written reports, and both the moderate and minimally guided groups performed equally well in the long term on an exam on the simulation topic, with significantly higher scores than the heavily guided group. The effects of guidance in pedagogical contexts have been observed in numerous studies without simulations, including evidence that student directed exploration can be productive and more thorough than instructor led instruction (Hawkins, 1974; Bonawitz et al., 2011).

This work draws on the emerging theoretical framework of implicit scaffolding, which combines and builds upon the theories of tool-mediated learning, tool design, and human computer interaction (Podolefsky et al., 2014). The PhET Interactive Simulations project at the University of Colorado Boulder provides students with interactive, implicitly-scaffolded tools for exploring science concepts through rapid inquiry cycles in a low-risk environment. Implicit scaffolding – guidance that is built into the design elements of the simulation to encourage students to productively explore with minimal directions (Paul et al., 2012; Lancaster et al., 2013; Moore et al., 2013; Podolefsky et al., 2014) – has been shown to support self-directed exploration and conceptual learning in an individual interview environment (Adams et al., 2009; Podolefsky et al., 2010). These findings led to our present investigation of how written guidance – i.e., an accompanying activity worksheet – impacts student engagement and exploration of a PhET simulation in a classroom setting.

There are many facets to understanding how teaching and learning materials influence students' educational experience. The research presented here focuses on student engagement – that is, students' interaction with the simulation during class – rather than measures of students' content learning. We approached this question by comparing the features in the simulation that students used and attended to for three different activities – each with a different level of written guidance.

Our specific research question was: How does the level of guidance affect engagement with a PhET simulation? We hypothesized that the implicit scaffolding in the simulation – including dynamic feedback and illuminating cases – would enable students with minimal written guidance to fully engage with the simulation in a classroom setting, consistent with Alfieri et al. (2011). It was also possible, however, that the lack of explicit guidance in the written activity could leave minimally guided students not knowing what to do (Kirschner et al., 2006) and lead to lower levels of engagement compared to students who received explicit written guidance on simulation use. Here, we examine the data to answer: whether students were more engaged with heavy or light guidance; whether students noticed important features in the simulation when not directed to them; whether students with heavy guidance explored other aspects of the simulation outside of their instructions; and finally, whether the minimally guided students explored more broadly but perhaps more shallowly, clicking through the simulation rapidly without attending to its content.

Methods

Data were collected from eleven recitation class periods on the topic of acid and base solutions. Students were organized into self-selected pairs, and all students worked on their own copy of the activity sheet for their class. Data consisted of simulation interactions (mouse clicks), classroom field notes, and a delayed redraw task administered during the following week.

Class description and participants

This study was conducted in the second semester of general chemistry at a large research university. The course included three 50-minute lectures and one 50-minute recitation period each week. (At this university, general chemistry laboratory is a separate course.) Course enrollment consisted of approximately 37% freshman, 36% sophomore, 13% junior, 11% senior/senior+ (students attending longer than 4 years), and 3% non-degree students. Declared student majors included life sciences (63%), physical sciences and mathematics (13%) engineering (7%), and various humanities and management majors (5%), with 8% of students undeclared. The simulation study was conducted over one week (5 days) across 11 recitation sections; an average of 20 students attended each section. Data were collected for 210 students, representing those students who both attended recitation and completed a consent form. These students represented 91% of total enrollment.

Informed consent was obtained in lecture prior to the simulation study. Researchers verbally described the simulation study in general terms, explaining that students' grades would not be affected by the study, that the course instructor would have no knowledge of students' choices to participate, and that participation would help inform and improve chemistry instruction for future students. Participating students signed and returned a paper consent form, and all students were provided an electronic copy of the form on their course management system. The simulation study was conducted in compliance with the relevant laws and institutional guidelines, and was approved by the Institutional Review Board (IRB) in a protocol for classroom simulation studies.

During the 50-minute recitations, students worked in pairs on university-owned laptop computers. In each recitation, nearly all groups consisted of two students (on occasion one or two students worked alone, or in a group of three); all students received their own copy of the activity sheet for their class. Students received participation credit for the recitation activity, per the usual practice for this course. No additional credit was assigned for participating in the research study. One graduate and one undergraduate student facilitated each recitation. As part of their regular weekly 1 hour preparatory meeting, the facilitators reviewed the activities and simulation, and discussed strategies such as encouraging students to take turns controlling the simulation and asking open-ended questions without providing simulation-specific guidance.

Simulation description

PhET's Acid–Base Solutions simulation (available at http://phet.colorado.edu/en/simulation/acid-base-solutions) was designed to support students in examining the role of solution concentration and strength for acids and bases. The main concepts addressed by the simulation include:

• Effects of strength and initial concentration on the pH and physical properties of acid and base solutions at equilibrium

• Coordination across macroscopic, microscopic, and symbolic representations (Johnstone, 1993) for acids and bases of varying strength and concentration

The simulation implicitly scaffolds student exploration, supporting students to construct an understanding of how strength and concentration affect pH, conductivity, and the equilibrium concentration of reaction species in each solution. The simulation uses two tabs. The Introduction tab (Fig. 1) includes five solutions – water plus four specific acid and base solutions with identical concentrations but different strengths; these solutions provide useful contrasting cases that exhibit key differences. The Custom Solution tab (Fig. 2) then allows the user to explore further by independently manipulating both concentration and strength with variable controls. Thus, through tabs, the simulation implicitly sequences the introduction of these two commonly challenging topics of strength and concentration (de Vos and Pilot, 2001).


image file: c4rp00009a-f1.tif
Fig. 1 The Introduction tab of Acid–Base Solutions, showing the molecular (top), quantitative (middle), and macroscopic (bottom) representations for a strong acid. When students select views and measurement tool options in the control panel, their choices are immediately reflected in the representations in the play area. Equilibrium concentration graphs (middle) and a conductivity meter (bottom) give students quantitative and qualitative means of comparing different solutions.

image file: c4rp00009a-f2.tif
Fig. 2 The Custom Solution tab of Acid–Base Solutions, showing the molecular view (top) and equilibrium concentration graph (bottom) for a weak base. Instead of the preset solutions used in the Introduction tab, here students are given two slider controls for a continuous investigation of changing initial concentration and acid or base strength.

For each tab, the placement and design of controls cue student interaction. On the Introduction tab, the groupings of radio button controls cue students to compare between different solutions. The views and measurement tools in the simulation facilitate students' coordination of the particulate (Molecules view), symbolic (chemical formulas and the reaction equation), and physical properties (pH and conductivity) of each solution. For example, in the Custom tab as students increase the initial concentration, the number of particles immediately and dynamically increases and the pH value changes. The simulation design also guides students' exploration through productive constraints; for example, the measurement tools appear poised above the beaker and can only be moved into and out of solution. Table 1 provides a list of simulation features and controls for both tabs of Acid–Base Solutions. The Acid–Base Solutions simulation's capacity to implicitly guide students in the ways described here was validated in individual interviews by university students who had not covered acids and bases in their general chemistry course.

Table 1 Simulation features and method of activation. The feature categorization columns indicate which features were counted as “exploratory” (E), “prompted” (P), or “neutral” (N) for each guidance condition. Default settings of the simulation on start-up are marked with *. Neutral features are shown in italics
Simulation features (and activation method) Feature categorization for…
Light guidance Moderate guidance Heavy guidance
Introduction tab N* N* N*
Strong acid (radio button) E P P
Weak acid (radio button) E P P
Strong base (radio button) E E E
Weak base (radio button) E E E
Water (radio button) N* N* N*
Molecules view (radio button) N* N* N*
Show solvent view (check box) E E E
Concentration bar graph view (radio button) E E P
Liquid view (radio button) E E E
pH meter (radio button) N* N* N*
pH paper (radio button) N N N
Conductivity tester (radio button) N N N
pH meter tool (drag into solution) E E P
pH paper tool (drag into solution) E E E
Conductivity meter (completed circuit) E E E
Custom solution tab E P P
Acid (radio button) N* N* N*
Concentration (slider control) E P P
Strong (radio button) E P P
Weak (radio button) N* N* N*
Weak strength (slider control) E P P
Base (radio button) E E E
Concentration (slider control) E E E
Strong (radio button) E E E
Weak (radio button) N* N* N*
Weak strength (slider control) E E E

Activity design

Each recitation section used one of three written activities, each designed with a different level of guidance, ranging from very open-ended prompts with multiple correct answers, to specific questions with single correct answers derived from specified parts of the simulation. These activities embody characteristics found in PhET activities developed and used by practicing teachers, and reflect the range of guidance styles in today's classrooms (PhET, 2014). The Light Guidance activity is described as lightly guided because it does not mention any controls in the simulation, and students' goal was to investigate and observe anything in the simulation that they thought related to pH. The Moderate Guidance activity is described as moderately guided because students were given the goal of answering open response questions about the concepts of strength and concentration in acids, but were given minimal explicit instructions on which controls to manipulate. In this way, students with moderate guidance were given freedom in how they used the simulation to carry out their investigations, but directed in the specific concepts they should investigate. The Heavy Guidance activity is described as heavily guided because the instructions told students to interact with the simulation in a specific way, with a prescribed sequence and flow. In this condition, there were no directions to explore or use the simulation in any way other than the set of instructions in the activity. In order to measure exploration in the Moderate and Heavy Guidance activities, these activities did not include prompts or questions for every feature in the simulation; features pertaining to base solutions were intentionally omitted from all explicit prompts. Example prompts from each activity, as well as the number of student groups and recitation sections are detailed below. Full activities are available in the ESI.†
Light guidance (LG). This condition had 4 recitation sections with 40 student groups. Students received a one-page activity sheet (Appendix A, ESI†) with the prompt “Explore the simulation with a partner. Record at least 10 observations below. Make sure to investigate all of the factors that affect the pH of a solution.” The rest of the page was blank for students to record their observations.
Moderate guidance (MG). This condition had 4 recitation sections with 40 student groups. Students received a two-page activity sheet (Appendix B, ESI†) containing eight prompts or questions, with approximately a quarter page of blank space provided for each instance where students were asked to record observations and answers. In Part I, students were instructed to “Explore the simulation with a partner. Record some of your observations below.” In Part II, students were prompted to “Use the Introduction tab to compare strong and weak acid solutions” and were asked two questions:

1. Describe all the ways that strong and weak acid solutions are similar and different, and explain why that makes sense using evidence from the simulation.

2. Does the pH of an acid solution measure the strength of the acid? Explain your reasoning.

Part III of the activity started on the second page: “Use the Custom Solution tab to explore acid concentration and strength.” Students were asked three questions:

1. Describe all the ways that the initial acid concentration affects the solution and the measured pH, and explain why that makes sense using evidence from the simulation.

2. Describe all the ways that the acid strength affects the solution and the measured pH, and explain why that makes sense using evidence from the simulation.

3. Reconsider your answer to [Part II, question 2]. Use evidence from the simulation to support your answer.

Heavy guidance (HG). This condition had 3 recitation sections with 33 student groups. Students received a four-page activity sheet (Appendix C, ESI†) with 40 numbered prompts and spaces, boxes, and tables to record data and answers. Students were given instructions on how to adjust simulation controls and what specifically to record on the activity sheet. In Part I, prompts guided students to use specific controls in the Introduction tab to display and record information about strong acids, including the reaction equation, the Molecules view, the Equilibrium Concentration bar graph, and the solution pH. For example:

4. In the “Solutions” section of the control panel, select “Strong Acid”

5. Record the equation. Strong Acid Equation: (space for answer)

6. In the “Views” section of the control panel, make sure “Molecules” is selected.

7. Describe the “Molecules” view in the box below.

8. In the “Views” section of the control panel, select “Equilibrium Concentration.”

9. Describe the “Equilibrium Concentration” view in the box below.

10. Dip the pH meter into the solution.

11. Record the pH. Strong Acid pH = (space for answer)…

On the second page, students were instructed to repeat these steps for weak acids, and then to “List two ways that strong and weak acid solutions are different.” Part II began on the third page, where students received step-by-step instructions to use the Custom Solution tab of the simulation to record ion concentration and pH data for strong acids. The activity provided a table for recording data for three initial concentration values specified by the activity.

2. In the “Solution” section of the control panel, make sure “Acid” is selected.

[…]

5. Set the “Initial Concentration” to 0.004 mol L−1.

6. Use the “Equilibrium Concentration” view to record the concentrations of [HA], [A−], and [H3O+] in the table below.

7. Dip the pH meter into solution and record pH in the table below….

Following the data table, students were asked “Based on the data you gathered, describe how the initial concentration of a strong acid affects the solution pH.” The data collection task was repeated for a similar table with three specified strength values for weak acid solutions. Students were again prompted, “Based on the data you gathered, describe how the strength of a weak acid affects the solution pH.” The last question in the activity was to “List two ways that a weak acid solution can have the same pH as a strong acid solution.”

Facilitation. Activity sheets were distributed in hard copy at the start of recitation and students were told they could keep the activity sheets. The typical structure for recitation in the course is students working in small groups on paper-based activities in a course booklet, with assistance from circulating student facilitators. In order to differentiate the effects of guidance level on simulation use, the facilitators were asked to conduct class in the normal way for content and conceptual questions and group discussions, but to not point out specific features in the simulation or give instructions on what students should click on.

Simulation interaction data

Students used a version of the Acid–Base Solutions simulation that was equipped to record a log of student actions for each student group's session (LG = 40, MG = 40, HG = 33; total = 113 logs). Student actions included mouse interactions such as clicks, drags, and click-and-dragging of objects and sliders in the simulation, as well as number entry from the keyboard. In this paper, the term “clicks” is used to describe all student interactions with the simulation. The student click data were analyzed to measure student interaction with the simulation over the course of the recitation period. Interaction was characterized by two measures: (1) what students clicked on – by comparison of the number of different interactive features that students used, where “use” of a feature is defined as clicking on its interactive control, or performing a series of steps to complete an action, such as moving the pH meter into solution, and (2) how often students clicked – measured by students' clicks per minute during simulation use, from time of first to time of last click, irrespective of beginning and end of class activity.
Simulation features. Table 1 lists all of the available simulation features – that is, the student actions possible with the simulation. For the analysis of interactive features used, we focused on 18 features in the simulation (denoted as “E” and “P”). To be categorized as using one of these 18 features, the student had to activate the feature either by clicking on it, or by clicking and dragging (for example, to take a pH reading by moving the meter into solution). The remaining 9 actions categorized as neutral, or “N”, were not included in the feature analysis (rows in italic).

The 18 features included in the analysis were characterized as “prompted” or “exploratory” for each condition, according to whether students were, or were not, explicitly prompted to use each one within their activity sheet. For example, if a student in the Heavy Guidance condition selected a chemical solution as instructed by their written activity (such as Strong Acid), and then clicked on a view option not mentioned in the activity (such as Show Solvent), these actions would constitute the use of two total features: one “prompted” feature (use of Strong Acid) and one “exploratory” feature (use of Show Solvent) in the simulation. Some of the 18 features were categorized differently between groups, based on each activity's instructions. The Light Guidance activity prompt did not refer to any particular features in the simulation, meaning all 18 of the analyzed features were in the “exploratory” category for this group. The numbers of simulation features in each category for the different guidance conditions are reported in Table 2.

Table 2 Simulation feature use by guidance condition. Possible features (PF), average number of features used (Used), standard deviation (SD), standard error of the mean (SEM), feature use ratio (ratio, calculated as a percentage from the number used of the number possible) and the use ratio SEM are given for each recitation activity guidance level. Feature use data were tested for normality using the Shapiro–Wilk test (Shapiro and Wilk, 1965), and p-values were calculated using contingency tables to compute the chi-squared statistic (see Appendix D, ESI). Note that the p-values obtained by this method are conservative compared to those obtained with parametric tests such as ANOVA (Press et al., 1992)
Guidance level Total features Exploratory features Prompted features
PF Used SD SEM Ratio (%) SEM (%) PF Used SD SEM Ratio (%) SEM (%) PF Used SD SEM Ratio (%) SEM (%)
a The Light Guidance condition did not have “Prompted” features in the written activity.
Light (n = 40) 18 15.8 2.6 0.4 88 2 18 15.8 2.6 0.4 88 2
Moderate (n = 40) 18 14.3 2.7 0.4 80 2 12 8.7 2.5 0.4 72 3 6 5.7 0.5 0.1 94 1
Heavy (n = 33) 18 9.0 1.8 0.3 50 2 10 1.0 1.8 0.3 10 3 8 8.0 0.0 0.0 100 0
χ 2 = 96.2, df = 20, p < 0.0005 χ 2 = 122.8, df = 20, p < 0.0005 χ 2 = 13.0, df = 4, p < 0.02

The 9 “neutral” features (Table 1, rows in italic) excluded from analysis were those available to students without click interactions (N*) or those that could be considered a pre-requisite action for a later click (N). For example, the opening default settings of the simulation such as the Introduction tab and Water could be viewed and investigated without being clicked. Similarly, the tool radio buttons, used to select tools prior to use, were excluded from analysis since this selection was a pre-requisite for tool use, which was counted and analyzed.

For the clicks-per-minute analysis, any click that represented a student's intent to interact with the simulation was counted. For example, if a student used any one of the 27 features listed in Table 1, or if they tried clicking the non-interactive magnifying glass (a background image), each of these actions would be counted as one click. Clicks on blank or empty background space in the simulation were not counted.

Classroom field notes

One of two researchers observed each recitation section and recorded field notes describing student interactions with the simulation, the activity, and the facilitators, as well as general classroom atmosphere. Field notes were compiled and reviewed for indications of students completing the simulation activity, as well as instances of off-topic conversation and actions, and served as a secondary source of data to inform interpretation of the student click data.

Redraw task

In lecture the week following recitation,‡ students were asked to recreate the simulation from memory on a page that read, “Last week, you used a PhET simulation in recitation. Did you use the simulation after the recitation? Circle YES or NO. Use the space below to draw as many details of the simulation as you can recall.” The rest of the page was blank. We structured this task similarly to the redraw task designed by Schwartz et al. (2011). Student drawings were coded for 17 features§ represented in the simulation (Table 3). “Drawn” features included all forms of representation: pictures, words, or options in a menu (as they were presented in the simulation). A coding scheme was created based on the features in the simulation, and a primary coder evaluated all student drawings (n = 197). The coding scheme was revised to clarify the codes for ambiguous representations (for example, circles representing molecules of an unspecified solution were coded as “Molecules View”, but not “Acid” or “Base”), and a secondary coder evaluated a set of 20 student drawings. These results were compared and clarification was added to the coding scheme until inter-rater agreement on a new set of 20 drawings was 95%. Nine student drawings were omitted from the analysis because the students either circled “Yes” to having used the simulation after recitation or did not circle either answer.
Table 3 Seventeen features from the simulation (solutions, views and tools) were coded from the student drawings. Values reflect the percentage of students that represented each feature, and the standard error of the mean. Features that students were not explicitly guided to use during the simulation activity are shown in bold text
Features drawn Light guidance (n = 66)/% Moderate guidance (n = 68)/% Heavy guidance (n = 54)/%
a It is possible that the low incidence of drawing the reaction equation in the Light and Moderate Guidance conditions was owing to the non-interactivity of this feature in the simulation; the high incidence in the Heavy Guidance condition is likely because the activity prompted students to record the equation.
“Exploratory” category for all groups
Base solution 80 ± 5 63 ± 6 46 ± 7
Strong base solution 35 ± 6 24 ± 5 9 ± 4
Weak base solution 38 ± 6 21 ± 5 13 ± 5
Show solvent view 23 ± 5 16 ± 4 6 ± 3
Liquid view 36 ± 6 26 ± 5 4 ± 3
pH paper tool 70 ± 6 51 ± 6 4 ± 3
Conductivity meter tool 76 ± 5 56 ± 6 4 ± 3
“Exploratory” for Light and Moderate, “Prompted” for Heavy
Bar graph view 36 ± 6 25 ± 5 56 ± 7
Reaction equationa 18 ± 5 10 ± 4 43 ± 7
“Exploratory” for Light only, “Prompted” for Moderate and Heavy
Strong acid solution 42 ± 6 38 ± 6 65 ± 7
Weak acid solution 55 ± 6 47 ± 6 72 ± 6
Solution strength 79 ± 5 79 ± 5 81 ± 5
Solution concentration 44 ± 6 71 ± 6 59 ± 7
Simulation default features (displayed on startup)
Acid solution 86 ± 4 88 ± 4 98 ± 2
Water 38 ± 6 26 ± 5 59 ± 7
Molecules view 88 ± 4 88 ± 4 85 ± 5
pH meter tool 82 ± 5 91 ± 3 81 ± 5

Results and discussion

As measures of how written guidance affects student engagement with a PhET simulation, we present results showing how many simulation features students used, whether student simulation use was more exploratory or more prompted in character, how much students interacted with the simulation, the rate of interaction over the class period, and measures of student attention to their interactions with the simulation. Simulation interaction data (mouse clicks) were used to analyze the features used and clicks over time, with classroom field notes and analysis of the redraw task supporting the conclusion that students were attending to their use of the simulation.

Engagement with the simulation

Simulation interaction data, including student clicks, were collected to address the duration, range, and amount of students' interaction with the simulation during the recitation activity. These data show clear distinctions across the Light, Moderate, and Heavy Guidance conditions. Here we examine the features that students used, the frequency of interaction, and whether or not these interactions were explicitly prompted by the activity.
Number of features used. An overall comparison of the number of features used shows significant differences (p < 0.0005) between guidance conditions (Fig. 3, Table 2). While all of these features were available to all groups using the simulation, students in the Light Guidance condition used nearly twice as many features as students in the Heavy Guidance condition.
image file: c4rp00009a-f3.tif
Fig. 3 The number of features used, shown as a percentage of all possible features in each feature category. The “All Features” graph shows the percentage of features used by each guidance condition for the set of 18 features that were analyzed. In the “exploratory” and “prompted” features graphs, these 18 features are divided between categories based on guidance condition (see Table 2 for feature category assignments). Category assignments were not identical between groups; for comparison, the percentage of features used in each category is shown. Error bars are the standard error of the mean.
Amount of exploration. The results in Fig. 3 show that while the written guidance in the Heavy Guidance activity assured the use of the prompted features mentioned in the activity instructions (only about acids), this level of guidance strongly discouraged students from exploring the other features of the simulation, such as the features dealing with bases. Examining the detailed log files, we found that only 11 of 33 Heavy Guidance groups clicked any “exploratory” features, and that the average number of “exploratory” features used by the Heavy Guidance students was strongly influenced by five groups that explored an additional 4–6 “exploratory” features at the end of their simulation use – most likely after they had finished their activity sheets, because the written activity did not call for these actions. In contrast, the Light and Moderate Guidance groups, both of whom had instructions to first explore the simulation and record any observations, used a great deal more of the features in the simulation throughout, including most of the “exploratory” features available to them.
Amount of interaction. The number of student mouse clicks per minute provides insight into the amount of student engagement with the simulation during the recitation activity. Fig. 4 shows a decrease in interaction with the simulation as the amount of written guidance in the activity increased. These data imply that the more open activity with fewer prompts invited more student interaction and exploration with the simulation – findings that are worthwhile to confirm for educators.
image file: c4rp00009a-f4.tif
Fig. 4 Clicks per minute during simulation use, averaged for each guidance level (LG = 7.6, MG = 5.4, HG = 2.2). These data represent interactions with all features of the simulation; error bars show standard error of the mean.

The clicks-per-minute data can be unpacked to examine trends in simulation use over time, such as the amount of time students interacted with the simulation and the average number of clicks per minute over the course of simulation use. A comparison of the amount of time students in each condition used the simulation (Fig. 5a) shows that all student groups interacted with the simulation for 10 minutes, with 90% interacting with the simulation for at least 15 minutes. After 15 minutes, the percentage of groups interacting with the simulation began to decrease for all guidance conditions. The average amount of time students spent using the simulation was comparable for the Light Guidance and Heavy Guidance conditions (21.3 min and 20.6 min respectively). From these data we see that groups in the Light Guidance condition devoted just as much time to interacting with the simulation as the Heavy Guidance group, indicating that less guidance did not result in lower engagement. While it is not clear why the Moderate Guidance group worked longer with the simulation (an average of 25.0 min), one possible contributor is that students in the Moderate Guidance condition were first instructed to explore the simulation, and then to work on their written prompts (whereas the Light Guidance group was only told to explore, and the Heavy Guidance group only received prompts). It is possible that combining these two types of tasks led to longer average use times with the simulation.


image file: c4rp00009a-f5.tif
Fig. 5 (a, top) Percentage of students using the simulation over time from first click to last click. The average amount of time to complete the activity was 20–25 minutes for all groups. (b, bottom) Average number of clicks per minute for groups using the simulation, starting with students' first click. As the number of groups using the simulation decreases (top), the average clicks per minute represent simulation use by fewer groups. Error bars show standard error of the mean for average clicks in each 1-minute increment. The clicks per minute data (bottom) were averaged over the number of groups using the simulation at each minute (top) within each guidance condition. The decrease in groups using the simulation is also reflected in the larger values for standard error of the mean starting at about 25 minutes.
Variance of click rate over time. In Fig. 5b, the number of clicks per minute varies substantially across the different guidance conditions, and can be understood in terms of the type of tasks students were pursuing. Initially, both the Light and Moderate Guidance groups were focused on freely exploring the simulation, as reflected in their similarly high click rates. The groups in the Heavy Guidance condition were clicking infrequently, since they were primarily focused on their extensive instructions and recording answers on their worksheets. Later (∼10 minutes) the Moderate Guidance condition became more engrossed in following the question prompts and their click rate slowed down, while the Heavy Guidance condition's click rate increased slightly (although remained the least) for the data collection portion of their activity. The Light Guidance condition continued clicking, and hence presumably exploring, at roughly a constant rate until the very end. The click rate for the Heavy Guidance condition increased after ∼25 minutes. At this time, most of the groups had completed the assigned activity, and a relatively small number of Heavy Guidance groups then took the opportunity to explore, once they were freed from the constraints of the activity.

Quality of student focus

An underlying goal of many (if not all) instructional activities is to engage students in thinking about and attending to the content and concepts at hand. The simulation interaction data indicate that students were clicking on and using the simulation, with the Light Guidance condition doing the most clicking on the most features, but these data do not prove that the students were attending to their interactions with the simulation. One observation that suggests that students were, in fact, paying attention is that the average number of clicks per minute for the Light Guidance condition is about 8, showing there is time for a considerable pause between each click for students to consider and process the resulting feedback from the simulation. Moreover, two additional data sources – the classroom field notes listing the amount of off-topic discussion and the delayed test of what students remembered about the simulation – show that the students in the Light Guidance condition (as well as those with Heavy and Moderate Guidance) were attending to the simulation during their interactions.
Classroom field notes. We analyzed the observers' field notes for instances of off-topic conversation during the simulation activity. Only one instance of off-topic discussion was observed per guidance condition before the completion of the activity. After students had completed their activities – typically about 20 minutes into the class period – many more off-topic discussions were observed and recorded in the field notes (5–8 instances total for each guidance condition). Because there was only one observer per classroom, this is only a sampling of student discussion; however, the small number recorded indicates that off-topic discussion was not prevalent during simulation use.
Redraw task. During lecture, in the week following the recitation activity, students were given ten minutes to draw what they remembered from the simulation (Fig. 6). Student memorization of the simulation features was not an instructional goal; rather, this instrument provides a delayed measure of the extent to which students had attended to the simulation. In Table 4, we observe that students in the Light Guidance condition drew more features than students in the Moderate and Heavy Guidance conditions (p < 0.05). The substantial number of features recalled by all groups indicates that essentially all students in all three guidance conditions were paying attention while interacting with the simulation.
image file: c4rp00009a-f6.tif
Fig. 6 Example student drawing showing sketches of the molecules view, solutions options, and the pH paper and conductivity meter tools.
Table 4 Average number of features drawn by each guidance condition. The p-value was calculated using a contingency table, similar to the simulation feature use data (see Appendix D, ESI)
Guidance condition (n = students) Avg # of features drawn (17 possible) Avg % of features drawn SD SEM
Light (n = 66) 9.26 54 3.40 0.98
Moderate (n = 68) 8.22 48 2.74 0.33
Heavy (n = 54) 7.85 46 2.11 0.29
χ 2 = 44.92, df = 30, p < 0.05

This conclusion is supported by comparing the specific features that students drew to the categories of features students clicked in each guidance condition. Table 3 gives the percentage of students in each group who included representations of various features from the simulation in their drawings. Reflecting the effects of guidance in the written activity, students remembered the features they clicked on: the Heavy Guidance students primarily drew the prompted features, while the Light Guidance students drew a more even distribution of the features in the simulation. Medium Guidance students, who spent some initial time exploring and then focused on a portion of the features in the simulation, drew more prompted features than the Light Guidance condition and more exploratory features than the Heavy Guidance condition. For example, base solutions were not referred to explicitly in any of the activity instructions; students in the Light Guidance group represented base solutions in their drawings with greater frequency (80 ± 5%) than Moderate Guidance (63 ± 6%) and Heavy Guidance (46 ± 7%) groups. Overall, these results indicate that students were attending to the simulation features they clicked on, and that they interacted with the features in the simulation in a meaningful way.

Implications for research and practice

In this study, we compared student engagement with an interactive simulation using three different written activities, each with a different level of explicit guidance. We observed that activity design in terms of guidance level can strongly influence student engagement with a PhET simulation – an educational tool specifically designed to support inquiry. Our results suggest that written guidance affects which simulation features students use and how much of the simulation they explore. The expectation for students implied by an activity's written structure and prompts can promote or inhibit student exploration of the simulation, as seen in the difference in student use of “exploratory” and “prompted” features within each guidance condition. Akaygun and Jones (2014) observed that in a classroom setting, guidance in the activity shifted students' focus – with more guidance resulting in stronger focus on the written activity rather than the simulation itself. Our study supports this finding, where explicit instructions shifted student attention to the directions and away from exploratory interaction with the simulation.

Implications for educators

The prevalence of heavily guided simulation activities used by practicing teachers in today's classrooms (PhET, 2014) suggests that some instructors have not fully considered the idea that the simulations provide guidance themselves, and can be used in ways that support more student-centered pedagogies. It is important that educators do not assume that students will explore all features of an interactive simulation regardless of the style and type of guidance in their written activity. Rather, the pedagogical approach built into the activity sheets strongly influences the nature of simulation use, and overall educational experience (Hennessy et al., 2006). We have shown that explicit guidance can significantly reduce the number of features that students click on, and will inhibit students' inclination to explore. When step-by-step instructions were provided, students tended to click on only those features that were included in the directions. Thus, by guiding students explicitly to the acid features, the activity effectively guided students away from the parts of the simulation that showed equally important content and concepts about base solutions. This finding suggests that it is either necessary to guide students explicitly to all features – which can be impractical for content-rich simulations – or that guidance in the activity must be structured in a way that will cue students to investigate the whole simulation (if desired). By allowing students the freedom to explore, with the affective and cognitive benefits that accompany this freedom, students examined and engaged with the simulation in ways that resulted in equivalent time on task and broader memory of the simulation content. This demonstrates that the implicit scaffolding built into the simulation effectively supports light guidance in an accompanying activity, with students noticing and using all of the important features in the simulation.

Scaffolded guidance – initially light guidance promoting exploration and gradually increasing toward specific learning goals – is our recommendation for giving students the opportunity to engage in the scientific practices of exploring and asking their own questions as they investigate the simulation, and to focus on the content and conceptual learning by answering questions that directly target topical learning goals. We further recommend that these later, topic-specific prompts continue to avoid direct instruction on simulation use. Instead, guidance should focus exploration in ways that continue to engage students' investigative abilities with the simulation such as answering challenge questions (“Find all the ways that …”), distinguishing hypotheses with simulation evidence, developing predictive models, and evaluating ideas. For additional information on teaching with PhET Simulations, see Moore et al. (in press). Indeed, Gardner et al., (2012) have presented a highly successful scaffolded guidance model where students first independently explored simulations, and then used the simulations to work on instructor-facilitated activities in small groups.

Instructors who typically use activities that begin with heavy instructions and later allow students to become more independent users of technology may want to carefully examine the implicit guidance in the educational tools at hand; in PhET simulations, a significant amount of implicit guidance is included in the simulation design, making the less-to-more guidance model a choice that is better supported by the evidence in this study.

Implications for researchers

With the growth of technology-enhanced teaching and learning in classroom practice, interactive simulations have become an important topic in chemical education research (Suits and Sanger, 2013). In studying the outcomes of using simulations for teaching chemistry, it is important to consider that simulations are part of an integrated system of resources that guide student learning: implicit guidance in the simulation, as well as explicit guidance from the written activity and the instructor influence how students interact and engage with simulations, and what students' expectations may be about how the simulations “should” be used. Even for simulations designed to support students' concept construction, a series of explicit instructions on what to click may undercut the affordances of the simulation by cueing students to use the simulation only as needed for generating specific answers, rather than as a means of conducting experiments and engaging in inquiry. It is important for researchers investigating the effects of learning with simulations to carefully consider the accompanying sources of guidance – especially procedural instructions (Chang et al., 2008) – and evaluate their potential to influence student perceptions, affect, and learning with the simulations being studied.

Conclusion

The process by which students engage with educational simulations is often not examined – with most studies instead focusing on learning outcomes. It is important, however, to understand the process by which students engage in order to inform simulation design, lesson design, and activity design. In this paper, we have shown that students provided with an interactive simulation and minimal guidance in a classroom setting actively engaged with the simulation and attended to the specific features of the simulation. Furthermore, we have shown that heavily guided activities can significantly decrease students' interaction and exploration of a simulation, limiting students to only use the features mentioned in the instructions. These results are important for the design of written activities to accompany simulations, and for understanding factors that support student practice in scientific inquiry. Questions for future study include how different levels of guidance can be optimized to support specific learning goals, what sequence of guidance fosters scientific practices and process skills, and how guidance from instructor facilitation and peers affects students’ engagement with the simulation and the activity.

Acknowledgements

This work would not have been possible without the valuable contributions of the entire PhET Team. We especially thank: Chris Malley as lead software developer of the Acid–Base Solutions simulation; Chris Malley and Sam Reid for developing simulation data collection and analysis; Cameron Barnhard and Jesse Garrison for assisting with data processing and coding; Noah Podolefsky for thoughtful discussions; and Carl Wieman for a thorough and critical reading of the draft manuscript. This work was funded in part by the Hewlett Foundation and the National Science Foundation DUE #1226321 and #0817582.

References

  1. Adams W. K., Reid S., LeMaster R., McKagan S. B., Perkins K. K., Dubson M. and Wieman C. E., (2008a), A study of educational simulations part I – engagement and learning, J. Interact. Learn. Res., 19, 397–419.
  2. Adams W. K., Reid S., LeMaster R., McKagan S. B., Perkins K. K., Dubson M. and Wieman C. E., (2008b), A study of educational simulations part II – interface design, J. Interact. Learn. Res., 19, 551–577.
  3. Adams W. K., Paulson A. and Wieman C. E., (2009), What levels of guidance promote engaged exploration with interactive simulations? Proceedings of the 2009 Physics Education Research Conference, AIP Press.
  4. Akaygun S. and Jones L. L., (2014), How does level of guidance affect understanding when students use a dynamic simulation of liquid–vapor equilibrium? in Devetak I. and Glazer S. A. (ed.), Active Learning and Understanding in the Chemistry Classroom, Springer, vol. XVII, p. 243.
  5. Alfieri L., Brooks P. B., Aldrich N. J. and Tenenbaum H. R., (2011), Does discovery-based instruction enhance learning? J. Educ. Psychol., 103(1), 1–18.
  6. Bonawitz E., Shafto P., Gweon H., Goodman N. D., Spelke E. and Schulz L., (2011), The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery, Cognition, 120, 322–330.
  7. Chang K.-E., Chen Y.-L., Lin H.-Y. and Sung Y.-T., (2008), Effects of learning support in simulation-based physics learning, Comput. Educ., 51, 1486–1498.
  8. Chang H.-Y. and Linn M. C., (2013), Scaffolding learning from molecular visualizations, J. Res. Sci. Teach., 50(7), 858–886.
  9. de Jong T. and van Joolingen W., (1998), Scientific discovery learning with computer simulations of concept domains, Rev. Educ. Res., 68, 179–201.
  10. de Vos W. and Pilot A., (2001), Acids and bases in layers: the stratal structure of an ancient topic, J. Chem Educ., 78, 494–499.
  11. Gardner T. Q., Kowalski S. E., Kowalski F. V., (2012), Interactive simulations coupled with real-time formative assessment to enhance student learning, Proceedings of the 2012 American Society for Engineering Education Annual Conference, AC 2012-5123.
  12. Gonzáles-Cruz J., Rodríguez-Sotres R. and Rodríguez-Peganos M., (2003), On the convenience of using a computer simulation to teach enzyme kinetics to undergraduate students with biological chemistry-related curricula, Biochem. Mol. Biol. Educ., 31, 93–101.
  13. Hawkins D., (1974), Messing about in science, in Hawkins D. (ed.), The Informed Vision, Essays on Learning and Human Nature, Agathon Press.
  14. Hennessy S., Deaney R. and Ruthven K., (2006), Situated expertise in integrating use of multimedia simulation into secondary science teaching, Int. J. Sci. Educ., 28 (7), 701–732.
  15. Johnstone A. H., (1993), The development of chemistry teaching: a changing response to changing demand, J. Chem. Educ., 70, 701–704.
  16. Khan S., (2011), New pedagogies on teaching science with computer simulations, J. Sci. Educ. Technol., 20, 215–232.
  17. Kirschner P. A., Sweller J. and Clark R. E., (2006), Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching, Educ. Psychol., 41, 75–86.
  18. Lancaster K., Moore E. B., Parson R. and Perkins K., (2013), Insights from using PhET's design principles for chemistry simulations, in Suits J. P. and Sanger M. J. (ed.), Pedagogic Roles of Animations and Simulations, ACS Symposium Series, Washington DC: American Chemical Society, vol. 1142, pp. 97–126.
  19. Lee H., Plass J. L. and Homer B. D., (2006), Optimizing cognitive load for learning from computer-based science simulations, J. Educ. Psychol., 98(4), 902–913.
  20. Moore E. B., Chamberlain J. M., Parson R. and Perkins K. K., (in press), PhET Interactive Simulations: Transformative Tools for Teaching Chemistry, J. Chem. Educ..
  21. Moore E. B., Herzog T. A. and Perkins K. K., (2013), Interactive simulations as implicit support for guided-inquiry, Chem. Educ. Res. Pract., 14, 257–268.
  22. Paul A., Podolefsky N. and Perkins K., (2012), Guiding without feeling guided: implicit scaffolding through interactive simulation design, Proceedings of the 2012 Physics Education Research Conference, AIP Press.
  23. PhET Interactive Simulations activity database, (2014), http://phet.colorado.edu/en/for-teachers/browse-activities, accessed April 21, 2014.
  24. Plass J. L., Homer B. D. and Hayward E. O., (2009), Design factors for educationally effective animations and simulations, J. Comput. High. Educ., 21, 31–61.
  25. Podolefsky N. S., Moore E. B. and Perkins K. K., (2014), Implicit scaffolding in interactive simulations: Design strategies to support multiple educational goals, http://arxiv.org/abs/1306.6544, accessed April 21, 2014.
  26. Podolefsky N. S., Perkins K. K. and Adams W. K., (2010), Factors promoting engaged exploration with computer simulations, Phys. Rev. ST Phys. Educ. Res., 6, 020117.
  27. Press W. H., Teukolsky S. A., Vetterling W. T. and Flannery B. P. (ed.), (1992), Chapter 14: Statistical description of data, in Numerical Recipes in C: The Art of Scientific Computing, 2nd edn, Cambridge University Press.
  28. Robinson W. R., (2000), A view of the science education research literature: scientific discovery learning with computer simulations, J. Chem. Educ., 77, 17–18.
  29. Rutten N., van Joolingen W. R. and van der Veen J. T., (2012), The learning effects of computer simulations in science education, Comput. Educ., 58, 136–153.
  30. Schwartz D. L., Chase C. C., Oppezzo M. A. and Chin D. B., (2011), Practicing versus inventing with contrasting cases: the effects of telling first on learning and transfer, J. Educ. Psychol., 103, 759–775.
  31. Shapiro S. S. and Wilk M. B., (1965), An analysis of variance test for normality (complete samples), Biometrika, 52, 591–611.
  32. Suits J. P. and Sanger M. J., (2013), Dynamic visualizations in chemistry courses, in Suits J. P. and Sanger M. J. (ed.), Pedagogic Roles of Animations and Simulations, ACS Symposium Series, Washington DC: American Chemical Society, vol. 1142, pp. 1–13.

Footnotes

† Electronic supplementary information (ESI) available: Written recitation activities used in the study for each guidance condition (Light Guidance, Moderate Guidance, Heavy Guidance), statistical analysis of simulation clicks and student drawing data, and activity construction tips. See DOI: 10.1039/c4rp00009a
‡ Students completed the redraw task 3–7 days after simulation use, depending on which day of the week they had their recitation section. Care was taken to distribute the three activity types over the days of the study week, as well as between morning, afternoon, and evening sections.
§ The set of features coded in the drawings analysis differed slightly from the features counted in the interaction analysis: some of the controls that could be counted individually in mouse click data were consolidated in the drawings coding scheme, and some of the non-interactive or neutral features in the simulation, such as the reaction equation, were included when coding the student drawings.
This journal is © The Royal Society of Chemistry 2014