Evaluating the effectiveness of the open-access ChemWiki resource as a replacement for traditional general chemistry textbooks

Abstract

Open educational resources (OERs) provide a potential alternative to costly textbooks and can allow content to be edited and adapted to a variety of classroom environments. At the University of California, Davis, the OER “ChemWiki” project, as part of the greater STEMWiki Hyperlibrary, was developed to supplant traditional post-secondary chemistry textbooks. The effectiveness of using this OER was assessed by comparing two general chemistry classes, one using ChemWiki and one using a traditional textbook, during the spring quarter of 2014. Student performance was measured using common midterms, final, and a pre/post content exam. We also employed surveys, the Colorado Learning Attitudes about Science Survey (CLASS) for Chemistry, and a weekly time-on-task survey to quantify students’ attitudes and study habits. The effectiveness of the ChemWiki compared to a traditional textbook was examined using multiple linear regression analysis with a standard non-inferiority testing framework. Results show that the performance of students who were assigned readings from the ChemWiki section was non-inferior to the performance of students in the section who were assigned readings from the traditional textbook, indicating that the ChemWiki does not substantially differ from the standard textbook in terms of student learning outcomes. The results from the surveys also suggest that the two classes were similar in their beliefs about chemistry and minimal overall study time. These results indicate that the ChemWiki is a viable cost-saving alternative to traditional textbooks.

---

Introduction

The growing options of educational pedagogies available for instructors to select from when developing and implementing courses includes an impressively rich range of scholastic opportunities. These include different instructional modalities such as Peer Led Team Learning (PLTL) (Eberlein et al., 2008), Process Oriented Guided Inquiry Learning (POGIL) (Eberlein et al., 2008), Massive Open Online Courses (MOOCs), Just in Time Teaching (JiTT) (Simkins and Maier, 2010), and flipped classes among others. Despite the potential benefits of such a wide range of instructional approaches, they present a distinct challenge in widespread implementation and dissemination since their application in classes are often mutually exclusive. For example, textbooks, as arguably the most important of such resources, are typically developed for use with a specific teaching approach and resources and may be of limited use for instructors unwilling to fully adopt the entire set of materials.

The all-or-nothing aspect of modern textbooks effectively hinders faculty from evaluating and eventually adopting multiple new teaching styles. This is especially detrimental since different topics presented in classes are often better suited for differing pedagogical approaches; hence a mixed approach may be most advantageous. Unfortunately, such mixed instructional approaches require flexible class resources and do not lend well to use of a single traditional textbook. The alternative of adopting multiple textbooks is cumbersome and unreasonably adds to the growing educational expenses for students. Clearly, textbooks must be developed to provide both flexibility and reduced cost to take full advantage of evolving educational opportunities.

The adoption of Open Education Resources (OERs) as alternatives to traditional textbooks addresses both flexibility and cost, while providing instructors the opportunity to easily evaluate and implement different educational approaches in the classroom without excessive investment of time and effort. Moreover, the free nature of OER textbooks helps to reduce growing education costs. Over the past several decades the extra cost of books and supplies in higher education has risen at a rate that is higher than medical expenses and the consumer price index (Perry, 2012). The College Board estimated the cost of textbooks and supplies for the 2013–2014 academic year was $1200 (Baum and Ma, 2014). This is an increase of approximately $300 from what was reported for the 2003–2004 academic year by the US Government Accountability Office (Office, 2005). Increases in textbook costs have risen at twice the rate of inflation (Office, 2005) and can impede access to higher education for many students with limited financial resources. It has become clear that an inexpensive alternative to high textbook costs should be developed, as was highlighted in the Higher Education Opportunity Act (2008).

While clearly beneficial in reducing educational costs, the growing interest in developing and using open-access textbook resources can be an effective mechanism to guide students toward a centralized “enhanced learning environment” capable of not only providing vetted textbook materials, but also self-consistently tracking student-specific activity and performance to identify weaknesses in study habits. The proposed Hyperlibrary project, initiated by the University of California, Davis in 2007, is one such resource that upon sufficient development can provide exceptional flexibility in addressing current and future educational needs. The Hyperlibrary is designed as a collaborative platform that enables dissemination and evaluation of new educational developments and approaches, with the opportunity to implement data-driven assessment of student learning and performance. The STEMWiki Hyperlibrary consists of multiple interconnected and independently operating STEMWiki hypertext applications (ChemWiki, BioWiki, MathWiki, StatWiki, GeoWiki, PhysWiki) that will be eventually augmented with ancillary homework and simulation applications as well as formative assessment modules. The Hyperlibrary's integrated and interdisciplinary nature empowers exceptional flexibility to address current and future educational needs and provides a platform for digital evaluation of cross-disciplinary learning. The content of these STEMWikis is both horizontally (across multiple fields) and vertically (across multiple levels of complexity) integrated within a massively interconnected network that provides, not just single textbooks, but an infinitely large Hyperlibrary through which interconnected STEM textbooks can be built. The ChemWiki is the primary and most developed component of the Hyperlibrary project (Rusay et al., 2011; Allen et al., 2015).

The focus of the STEMWikis is to develop and disseminate viable online post-secondary textbook alternatives within a centrally-integrated OER environment. From these components, we will build a combined assessment infrastructure that tracks and correlates use of individual Wiki-based textbooks with simulations, homework activity and exam performance with a goal of identifying and tracking student strengths and weakness for their benefit across multiple STEM curricula. This is enabled by the broad scope vertical and horizontal nature of the Hyperlibrary.

While traditional textbook construction has followed a simple, yet proven, paradigm whereby an expert (or small group of experts) with advanced training and experiences apply their collective education and expertise towards constructing the required content, the recent success of the OER Wikipedia project has demonstrated that alternative approaches can be similarly powerful with “crowd-sourced” development approaches involving substituting significant effort of a few number of experts with modest effort of many students and educators (at a range of capacities). A clear benefit of such a parallelized approach is that content can be rapidly constructed and updated. The development of the STEMWiki Hyperlibrary follows such an approach and is spearheaded by a consortium of students and faculty over multiple campuses and countries. Development entails collecting, integrating, vetting, and building open-access content within an extensively hyperlinked infrastructure consisting of independent pages containing well-defined concepts (e.g. “Units”, “Chapters”, “Sections” etc.) that are typically written, edited, and vetted by multiple authors.

A well-functioning textbook (whether hyper- or conventional) is much more than just a series of reference topics found in encyclopedias or Wikipedia, but must address additional aspects: (1) an established flow between previously discussed, current and future content and (2) a complementary set of questions to aid student internalization of the text material. Key to the utility of the ChemWiki is its intrinsic flexibility necessary to suitably address these aspects. All Modules containing information are contained in the Core (Fig. 1) and “Wikitexts” are individually constructed for specific classes by creating a hyperlinked structure to the Core Modules. This provides a powerful flexibility in introducing and removing content without affecting other concurrently operating classes and provides the flexibility for instructors to construct Wikitexts that best suit their needs (e.g., ignoring non-integral topics). Each Module contains metadata that outlines the recommended Modules necessary for students to have read prior to the Module to receive a full understanding of the content contained therein.

Fig. 1 Illustration of how the Core/Wikitext enables the flexible design of a variety of Wikitexts for courses at all levels of instruction and subfields; numbers are for UCD chemistry courses. Different modules coexist addressing the same topic, but at different levels allowing for addressing classes simultaneously.

Wikis provide a simple way to collaboratively generate, and publish reusable on-line content (O'Neill, 2005). In addition to providing up-to-date, peer reviewed, affordable and convenient content, an important pedagogical value of Wikis is how they support students' use of content to achieve learning objectives (Buffa, 2006). Grounded in social constructivism (Parker, 2007), Wikis allow learners to cooperatively construct and organize knowledge. Yang (1996) presented guidelines for incorporating constructivist theories into the design of a hypertext (or hypermedia) document and further evidence has shown that the architecture of hypertext is well-aligned with constructivist learning theories (Shorb and Moore, 2010). Because of their flexible and modular-based approach to instruction, Wikis provide an important alternative to the “one-size-fits-all” approach to instruction where content is presented in a static prepackaged manner. Next generation learning systems promote personalized, flexible, and interactive learning experiences that can take advantage of the inherent Wiki flexibility. Studies have found that students and faculty believe that OERs can be used as an effective classroom resource (Bliss et al., 2013; Allen and Seaman, 2014) and that some students respond more favorably to an OER (Lindshield and Adhikari, 2013). Furthermore, students are generally open to using free online resources and can save a significant amount of money by using them (Bliss et al., 2013; Hilton et al., 2013). Despite the positive attitudes toward OERs, higher education has been slow to adopt OERs and other online learning systems (Bacow et al., 2012; Allen and Seaman, 2014). Research suggests that this may be due to a lack of awareness slowing OER adoption (Allen and Seaman, 2014).

Additional research has focused on assessing the impact of OERs on student performance outcomes. A comprehensive multi-university study of Carnegie Mellon's Online Learning Initiative (OLI) in statistics courses showed that OLI students perform as well as students using traditional textbooks in both overall course performance and in class assessments (Lovett et al., 2008; Bowen et al., 2012). Work done by Hilton et al. in mathematics (Hilton et al., 2013) and psychology (Hilton and Laman, 2012) showed the students using OERs performed as well as those using traditional textbooks. In a study of science classes at public secondary schools, Robinson et al. found that students using OERs performed slightly better (Robinson et al., 2014).

Methodology

The purpose of this study was to assess the viability of using the ChemWiki to replace standard textbooks as the primary resource for an undergraduate general chemistry class. The study was designed to compare two Chem 2C (third-quarter introductory) classes: a control class that used the “General Chemistry Principles and Modern Applications” textbook by Petrucci et al. as the primary resource and an experimental class that used the ChemWiki as their primary resource. The performance, resource usage, and attitudes of the students were measured using in-class assessments and surveys. The students in each class were informed about the study and were not required to participate in the non-class specific assessments. After the data was collected and compiled we removed all personal information and utilized anonymous personal identification numbers. A proposal for the study was submitted to the internal review board and was deemed exempt from IRB regulation, but passed IRB review for human subjects research. All instruments used as part of this study, as well as the individual study components were reviewed and approved by the IRB.

Class structure

The ChemWiki comparison study occurred during the spring quarter (SQ) of 2014 in Chemistry 2C, the final course in UC Davis's three quarter general chemistry sequence. There were 448 students in the SQ Control class and 478 students in the SQ ChemWiki class. Both classes had three 50 minute lectures per week along with a three-hour lab and a one-hour discussion (recitation). The same instructor taught the lecture for both classes, back to back, to remove any instructor bias. Each teaching assistant taught one lab and discussion from each class to further remove any bias. Students in the SQ Wiki class were required to log into the ChemWiki via password in order to prevent students from the SQ Control class from using the ChemWiki resources and to help track the usage of the ChemWiki throughout the quarter.

Since the initial SQ ChemWiki experiment, the Chemwiki has been used as the primary resource for two additional classes at UC Davis: one class of Chem 2C taught during summer school 2014 and one class of Chem 2A, first quarter general chemistry, taught during the fall quarter of 2014. These two courses were not taught simultaneously with a control class, but data from these classes are presented and compared to other general chemistry courses taught at the same time. A summary of the included classes can be found in Table 1. The majority of the data presented in this paper are from the initial SQ 2014 experiment.

Table 1 Overview of classes involved in the ChemWiki assessment

Class	Year	Quarter	Instructor	Primary resource	Class size	Designation
Chem 2C	2014	Spring	A	ChemWiki	478	SQ Wiki
Chem 2C		Spring	A	Textbook	448	SQ Control
Chem 2C		Summer	A	ChemWiki	210	SS Wiki
Chem 2A		Fall	B	ChemWiki	489	FQ Wiki

---

Pre/post assessment exam

Prior to the start of the quarter, all instructors teaching chemistry agreed upon the same pre/post “standardized” assessment exam, an internally developed a 50 minute, 35 question multiple choice exam to determine the overall learning gains of the students after taking each quarter of general chemistry at UC Davis. The exam covered material that would be taught throughout the quarter. Two versions of the exam were written, one to be given at the beginning of the quarter and one at the end. Alternate form equivalency was established between the two versions of the exam (test form A mean = 19.44, test form B mean = 21.00, t₄₈ = −1.07, p = 0.290), and only minor wording changes were made since the original equivalency study.

Students took the course-specific pre-test during the first lab for Chem 2A and in the first discussion/recitation for Chem 2C. The course-specific post-test was embedded into the two-hour final exam for both Chem 2A and 2C. After the students took the final exam, normalized learning gains (NLG) were calculated using the following equation (Hake, 1998):

NLG = (%Post − %Pre)/(100 − %Pre)

Midterms and final

In the SQ 2014 experiment, two 50 minute midterms were given during the fourth and eighth week of the course in the normal lecture time, and a two-hour final was administered at the end of the course. The same midterms and final were used for both classes. The same teaching assistants graded the midterms and final at the same time, using a common grading rubric. Students' scores on the two midterms and the final were summed to calculate total exam points, a measure of their overall course performance. Course grade was used as an additional measure of overall performance. Midterm 1 and 2 each constituted 25%, the final was 40%, and the lab was 10% of the overall grade.

Weekly time-on-task survey

Each week, the students took a survey referred to as the “Weekly Time-on-Task Survey.” The purpose of the survey was to gather self-reported information on how many hours the students were using their primary resource each week and the number of hours they spent in lecture and on any secondary resources. It was also used to determine if and how students in the SQ ChemWiki class were using the textbook.

CLASS-Chem survey

The Colorado Learning Attitudes about Science Survey (CLASS) chemistry survey was given at the beginning and end of the quarters to measure if students' attitudes changed as a result of the primary resource they used for class. The CLASS Chem assessment is a 50-question survey that asks students to respond to statements about their beliefs on learning chemistry, content of chemistry, structure of chemistry, and connection of chemistry to the outside world (Barbera et al., 2008). The favorability of the student to chemistry in the ten different categories is based on how closely a student responds to an expert (Adams et al., 2006). By giving the survey at both the beginning and end of the quarter we can investigate whether or not the ChemWiki differentially impacts a student's favorability towards the subject and learning of chemistry.

Statistical analysis

To determine whether using the ChemWiki as the primary course resource resulted in comparable student achievement as the utilization of the standard textbook, data were analyzed using multiple regression with a non-inferiority framework. Because the purpose of this study was to determine whether the ChemWiki was an adequate substitute for the traditional textbook, the traditional comparative test in which the null hypothesis is that there is no difference between the treatments was not sufficient. In traditional comparative tests, the goal is to establish that there is a difference between treatments and the burden of proof relies on a difference that is large enough to be unlikely due to chance. If this is the case, the null hypothesis can be rejected and it can be said that the two treatments are statistically significantly different. In contrast, when the difference in treatments is small, it is not possible to determine whether the small difference is due to chance or whether the difference is due to the treatment. In this case, we fail to reject the null hypothesis. However this does not indicate that that the two treatments are equivalent, just as we cannot rule out that the difference is due to chance and that they are not statistically significantly different (Walker and Nowacki, 2011).

Non-inferiority testing is a method used to test whether a new product or intervention (e.g., the ChemWiki) is a suitable substitute for an existing competitor that is a standard acceptable practice (e.g., the textbook; (Allen and Seaman, 2007)). In non-inferiority testing, an equivalence margin (−δ,δ) is selected based on the maximum difference that one would be willing to accept as a difference between the traditional treatment and the new treatment (Allen and Seaman, 2007). The 90% confidence interval associated with the difference in treatment effects is then calculated, in this case using multiple regression. If the confidence interval includes zero and the lower bound is not less than the lower non-inferiority boundary, it can be concluded that the new treatment is non-inferior. A 90% confidence interval is used because non-inferiority testing utilizes a two one-sided test (1 − 2α) × 100% (Allen and Seaman, 2007), so the confidence interval yields a 0.05 significance level for testing equivalence.

For the purposes of this study, we chose the equivalence margin to be plus or minus 2% because this is the typical grade interval in the introductory chemistry course (i.e., the span for a B is from 83–87) and therefore, a difference of 2% in either direction would not impact a student grade, assuming their grade was right in the middle of a grade span. For example, for a student who earned a score of 85%, we would accept a score of as low as 83% to be non-inferior and a score of 87% to be non-superior.

Multiple regression with a non-inferiority framework was used to estimate the effect of being in the ChemWiki section on midterm 1, midterm 2, final exam score, and total course points. In each model, the dependent variable was modeled as a function of a binary variable indicating whether or not the student was in the ChemWiki section, as well as a series of covariates to control for differences in students' prior achievement and demographic characteristics. Table 2 shows the covariates that were included in the model and the descriptive statistics for the students in the ChemWiki section and the traditional section. In the models, all continuous variables were centered. There were no statistically significant differences between the groups on any of the covariates included in the model.

Table 2 Covariates

	ChemWiki (n = 377)	Textbook (n = 348)
Mean pretest score	10.57	10.56
First generation (%)	36	37
Low income (%)	26	25
Underrepresented minority URM (%)	18	17
Male (%)	38	36
Transfer (%)	4	4
STEM major (%)	91	90
Mean prior units completed	36.08	34.54
Mean SAT total score	1866.00	1848.00

---

Students who were missing data on one or more variables included in the model (n = 201), with the exception of SAT scores, were dropped from the analytic sample. Because SAT scores are not required for transfer students, a large proportion of students were missing this variable. Therefore, scores were imputed using multiple imputation methods for students missing this variable. Students who were excluded from the analytic sample were not significantly different from those included on any of the covariates included in the model, with the exception of number of previous units completed (analytic sample mean = 42.27, excluded sample = 35.34, t₉₂₄ = 3.65, p < 0.000).

Normalized learning gains were compared using independent samples t-tests, with a non-inferiority framework. A two-way repeated measures analysis of variance (ANOVA) was used to examine whether the changes in student's responses to the CLASS Chem differed by the type resource used. Finally, responses to the time on task survey were analyzed using independent samples t-tests to determine whether self-reported time on task differed between students in the ChemWiki and textbook section.

Results

Non-inferiority regression results

Tables 3 show the results of the multiple regression analyses examining the difference in the achievement of students in the ChemWiki and textbook sections, controlling for students' characteristics and prior achievement. The results of the multiple regression analyses indicate that there was no statistically significant differences in the performance of students in the ChemWiki and textbook sections on the post-test (β = 0.09, p = 0.913), midterm 1 (β = −1.45, p = 0.154), midterm 2 (β = 0.34, p = 0.690), final exam points (β = −0.01, p = 0.989), total exam points (β = −0.330, p = 0.653), or course grade (β = −0.48, p = 0.499). Furthermore, for all dependent variables with the exception of midterm 1, non-inferiority of the ChemWiki was established because the 90% confidence interval was contained within the equivalency margin (δ = ±2%; see Fig. 2). For example, after controlling for all covariates, the average total exam points in the textbook class was 69.80% and the average in the ChemWiki class was 69.47%, a difference of −0.33 percentage points. The 90% confidence interval for the difference in total exam points was (−1.53, 0.87) and, since this interval is included in the (−2, 2) equivalency margin, performance in the ChemWiki class is non-inferior to performance in the textbook course.

Table 3 Ordinary least squares regression model estimating effects of ChemWiki on the post score utilizing a non-inferiority framework

Variable	Post-test	Midterm 1	Midterm 2	Final	Total exam points	Course grade
*p < 0.05, **p < 0.01, ***p < 0.001. Note: All dependent variables are in percentage points; the equivalency margin for non-inferiority is ±2%.a SAT total was imputed using multiple imputation methods.
Constant	70.37***	63.89***	75.84***	38.39***	69.80***	75.05***
ChemWiki (90% CI)	0.09 (−1.22, 1.39)	−1.45 (−3.12, 0.22)	0.34 (−1.05, 1.73)	−0.01 (−0.72, 0.71)	−0.33 (−1.53, 0.87)	−0.48 (−1.66, 0.69)
Pre assessment	0.43***	0.53***	0.35***	0.25***	0.45***	0.42***
First generation	−1.75	−1.97	−2.04	−0.43	−1.47	−1.61
Low income	1.40	−0.16	0.88	−0.09	0.13	0.41
URM	−0.65	−1.83	−0.39	−0.66	−1.16	−0.63
Male	2.59**	−1.24	1.67	1.11*	0.98	0.53
Previous units	−0.05**	−0.12***	−0.03	−0.05***	−0.08***	−0.09***
SAT total^a	0.01***	0.01	0.01*	0.01**	0.01**	0.01**
STEM major	0.92	2.56	0.55	1.00	1.68	1.52
Transfer	−0.90	−6.54*	−4.29	−2.47*	−5.03*	−4.20*
R ^2	0.26	0.24	0.16	0.30	0.30	0.28

---

Fig. 2 90% confidence interval of the estimated effect of the ChemWiki on measures of student achievement displayed with the non-inferiority margin.

The only outcome for which we failed to establish non-inferiority was midterm 1. After controlling for all covariates, the average score on midterm 1 in the textbook was 63.89 and the average in the ChemWiki was 62.45, a difference of −1.45 percentage points. The 90% confidence interval for the difference in midterm 1 was (−3.12, 0.22) which falls outside the (−2, 2) equivalency margin, therefore, we fail to establish non-inferiority for this outcome. Fig. 2 shows the confidence intervals for each outcome variable displayed on the equivalency margin. In this figure, it is apparent that the only confidence interval that goes beyond the equivalency margin is the confidence interval associated with the effect on midterm 1. Because the confidence interval for midterm 1 extends beyond the lower bounds of the equivalency margin, we fail to establish non-inferiority. However, none of the other confidence intervals extend beyond the equivalency margin indicating that the ChemWiki is non-inferior for all other measures of students' achievement.

Normalized learning gains

The average normalized learning gain was 57.83% in the ChemWiki class and 57.58% in the textbook class, a difference of 0.25 (t₇₂₃ = 0.2, p = 0.842). The associated confidence interval is (−1.804, 2.303) which is within the equivalency margin indicating that the normalized learning gains made by student in the ChemWiki class were not inferior to those made by students in the textbook class.

CLASS Chem survey

A two-way repeated measures analysis of variance (ANOVA) was used to examine whether the change in students' responses on the CLASS Chemistry from pre to post differed by class. Results indicated that there was a statistically significant main effect of time (F_1,213 = 4.71, p = 0.031), indicating that, on average across both classes, students' overall scores on the CLASS Chemistry improved by 2.65 points. The interaction effect between class and time was not significant (F_1,213 = 0.052, p = 0.821) for the CLASS Chemistry Overall Score, indicating that both groups made similar changes over the course of the quarter.

Time on task

Independent samples t-tests showed statistically significant differences in the study habits of the students in the ChemWiki and textbook classes. Students in the ChemWiki section self-reported spending more time weekly reading assigned material (M = 2.03 h vs. 1.64 h; t₇₂₅ = −2.2, p = 0.024) and more overall time on task (M = 3.73 h vs. 3.36 h; t₇₂₅ = 2.3, p = 0.021), but spent less time attending their assigned lecture (M = 2.42 h vs. 2.56 h; t₇₂₅ = −2.7, p = 0.006) and less time using non-assigned reading (M = 0.72 h vs. 0.92 h; t₇₂₅ = 4.3, p < 0.001). There were no statistically significant differences in the amount of time student in the ChemWiki and textbook classes reported spending on practice problems (t₇₂₅ = 0.06, p = 0.949), the percentage of time they spent working with friends (t₇₂₅ = −0.52, p = 0.598), the number of lectures they were not registered for that they attended (t₇₂₅ = 1.48, p = 0.137), or the number of hours of private tutoring they received. Furthermore, results from a multiple regression analysis indicate that, compared to students in the textbook class, students in the ChemWiki section reported spending on average of 0.41 more hours studying, after controlling for prior achievement and demographic characteristics (β = 0.410, p = 0.008).

Outcome difference based on ChemWiki use

Students within the ChemWiki class were divided into quartiles; the bottom quartile was considered “Low Wiki Users” and the top quartile was considered “High Wiki Users.” An independent samples t-test indicated that, on average, High Wiki users tended to receive higher course grades (M = 79.76) than low Wiki users (M = 71.43, t₂₅₅ = −4.723, p < 0.000). This difference continued to be statistically significant after using multiple regression to control for students' prior achievement and demographic characteristics (β = 7.809, p < 0.000), with Low Wiki Users receiving an average grade of 71.91 and High Wiki Users receiving an average grade of 79.71.

Discussion

When accounting for incoming student characteristics (Table 2), differences in the performance measurements employed during the spring quarter study were not statistically significant. When the overall performance of the two classes was compared, using course grades and normalized learning gains, no statistically significant differences were observed, demonstrating that the two classes had comparable overall performance. The results from the multiple regression models (Table 3) clearly indicate that the ChemWiki did not significantly alter student performance. We failed to detect a significant difference between the ChemWiki and the traditional textbook classes using a multiple regression model. Therefore, we used non-inferiority testing to determine if the classes had similar performance.

After applying our non-inferiority criteria, ±2% equivalency margin on the estimates from the multiple linear regression models, we found that the ChemWiki class was non-inferior in all but one performance measure used throughout the study (Fig. 2). Only the first midterm failed to establish non-inferiority but the difference in performance across the two classes was not statistically significant. Midterm 1 was given during the fourth week of the week of the class and it was the first time students had taken the instructors exam. Therefore, it is possible that the students were not acclimated to the ChemWiki, which led to a slight, although not statistically significant, variation in student performance on midterm 1. This may explain why midterm 1 did not meet our non-inferiority criteria. Although the ChemWiki class did not pass our non-inferior testing criteria for midterm 1, the ChemWiki class was found to be non-inferior in areas of overall course and test performance. These results show that the ChemWiki is not inferior to the use of a traditional textbook, providing compelling evidence that the ChemWiki is a viable alternative to the costly traditional textbook.

The results from the time-on-task survey (Table 4) show that the students in the ChemWiki group reported spending more time interacting with their primary resource, spent less time using secondary/non-assigned reading, and overall spent more time on task than the control group. Although the ChemWiki group spent approximately 25 min more per week studying than the control group, we did not observe an increase in performance. In a study done by Bliss et al. the number of students using an OER daily increased but there was not a significant overall change in student performance (Bliss et al., 2013). It is possible that an increase of half an hour in study time is not sufficient to significantly impact student learning. It could also be that the students are spending slightly more time using the ChemWiki because they are following links to non-course related material. The integrated nature of the ChemWiki with other STEMWikis allows for students to explore subjects that are linked to the course specific pages but not covered by the class. A student could spend more time investigating these different subjects and therefore more time on the ChemWiki than if they were to use the traditional textbook. We currently cannot determine when and how often a student goes to non-course specific pages but it is something we plan to do in the future.

Table 4 Comparison of the mean time on task for students in the ChemWiki and textbook classes

	ChemWiki (n = 377)	Textbook (n = 348)
*p < 0.05, **p < 0.01, ***p < 0.001.a Time on task is a composite variable created by summing the average amount of hours a student spend reading recommended material, practicing homework problems, and “non-assigned reading materials.
Hours spent reading course materials	2.03***	1.64***
Hours spent on practice problems	1.53	1.52
Percent of time spent working with friends	13.85	14.56
Average number of weekly lectures attended	2.42**	2.56**
Average number of weekly lectures attended NOT registered	0.20	0.15
Hours of private tutoring a week	0.17	0.20
Hours spent using non-assigned reading	0.74*	0.92*
Time on rask^a	3.73*	3.36*

---

Somewhat alarming is the total time (about 3.5 h per week) students reported studying in both classes. On average students are studying about 1.2 hours per lecture hour outside of class where incoming students at UC Davis are expected to study approximately 2 to 3 hours per lecture hour outside of class. When we looked at high ChemWiki users versus low ChemWiki users we found that high ChemWiki users had course grade that were approximately 7.8% higher than low ChemWiki users (Table 5). Although these estimates are corrected for incoming performance characteristics, it is important to note that these findings are still correlational and we cannot say that the high ChemWiki use caused higher course grades.

Table 5 Low vs. high Wiki user descriptives

	Low Wiki users (n = 133)	High Wiki users (n = 132)
*p < 0.05, **p < 0.01, ***p < 0.001.a SAT total was imputed using multiple imputation methods.
Wiki page views	77.43***	414.64***
First generation (%)	36	33
Low income (%)	26	17
URM (%)	20	20
Male (%)	45*	32*
Transfer (%)	2	4
STEM major (%)	90	89
Final course score	71.43***	79.76***
SAT total^a	1871	1848

---

Students in both groups reported spending less than one hour a week on non-assigned (secondary) resources. The time spent on secondary resources accounted for about 20% of the ChemWiki class and 27% of the Control class' overall time on task. The survey was also used to account for students in the ChemWiki class that may have used the textbook as a secondary resource. Using the time on task survey we also found that 14.6% of the students in the ChemWiki section reported using the textbook at least once, while 26.6% of the students in the Textbook section reported using the ChemWiki at least once. Further, we compared students in the ChemWiki section to students in the Textbook section who reported using their primary resources >70% and >80% to determine if there was an effect of using multiple resources. Table 6 shows the average course grade and post exam performance for these two groups of students. In both instances for students who used the primary resource >70% and >80% of the time there wasn't a statistically significant difference in the average performance. The ChemiWiki class' performance also meets our non-inferiority testing model when comparing the course grade and post exam performance of the two primary resource groups. Therefore we believe that the overall comparison of the two classes is valid because of the small percentage of students in the ChemiWiki class using the textbook and the similarity in performance of the two groups who reported using their primary resource >70% and >80% of the time.

Table 6 Comparison of course grade and post exam performance of students who reported using their primary resource >70% and >80% of the time

	Course grade	Post%
	Mean C.I. 90%	Mean C.I. 90%
70% Wiki (n = 107)	75.37 ± 4.76	71.08 ± 4.37
70% Book (n = 97)	76.37 ± 4.52	73.13 ± 4.48
80% Wiki (n = 64)	75.16 ± 6.42	71.47 ± 5.52
80% Book (n = 53)	75.84 ± 6.52	70.81 ± 6.01

---

One way to potentially improve the amount of time students interact with the material is to increase the amount of collaborations between students. Although this was not explicitly investigated in this study, OERs, wikis specifically, offer an excellent platform for students to collaborate and contribute to group projects (Rasmussen et al., 2013). The ChemWiki itself is a large collaboration between students, faculty, and other contributing authors. Approximately one third of the content has been generated by students and pages have been developed as class projects in undergraduate and graduate classes at UC Davis and other locations. Work done at other institutions and on other wiki platforms has shown that collaborations on wikis can improve student engagement, learning, and writing proficiency (Tsai et al., 2011; Rasmussen et al., 2013; Pence and Pence, 2015). We did not use the Chemwiki as a collaborative platform in this study because group activities are not typically incorporated into the typical UC Davis general chemistry class and we were investigating the ChemWiki's ability to act as a standalone resource. However we plan to use the ChemWiki in the future to incorporate group projects and collaborations because of its ability to easily facilitate collaborative projects.

We used the CLASS Chem survey to investigate how the ChemWiki may impact student perceptions of chemistry. The comparison of the two classes showed that there were no statistically significant differences. Unfortunately, the way the survey was given reduced the overall pre and post response rate of the two classes to about 27% of the total students. However, the students that did take the survey are statistically representative of each class. The overall change in attitudes was similar between the two groups indicating that the two resources change a student's beliefs about chemistry in a similar way. The survey itself does not have questions targeted specifically to the resource that the student is interacting with, therefore we believe that the main effect on a student's opinions about chemistry is the expectations of the instructor and overall classroom experience. It is possible that the resource has the ability to change how a student connects chemistry to the real world, but we did not see any differences in this category of the CLASS Chem survey. However, it is possible for us to easily bring in more real world connections by adapting the ChemWiki content to try and improve the student's beliefs in this area, something that would take a new edition and printing of a traditional textbook.

Work done by Bliss and colleagues has found that students have an overall positive attitude towards OERs and enjoy using the resources online (Bliss et al., 2013). We did not specifically measure the student's attitudes towards the ChemWiki or textbook, but the end of the quarter course evaluations added some insights into the student's attitudes towards the ChemWiki. A couple representative quotes from student's comments show that the ChemWiki was valuable to the students and easy to navigate: “ChemWiki was incredibly well organized and valuable to me” and “I enjoyed the chemwiki textbook and resources and feel that it helped me in the course”. Some students in the textbook class commented that they wish they had more access to a ChemWiki testbank. There was some indication of errors in the ChemWiki and some frustrations with the practice problems used for the class, however because the ChemWiki is a dynamic text we easily addressed these errors the subsequent quarter. Moreover, we plan to build and integrate a more comprehensive homework/practice problem system into the project.

The application of the ChemWiki in a fall quarter Chem 2A class resulted in no significant difference on the common final (β = 0.008, p = 0.298), between one section that taught using the ChemWiki as the primary teaching resource and the other sections that used a traditional textbook, the ChemWiki was also used as the primary teaching resource during Chem 2C 2014 summer session. Because we did not have a control class in these quarters, these finding have low internal validity. However, we believe that because the students in these courses performed on par with students in other sections, both during the same quarters and previous quarters, this provides further evidence that the ChemWiki can be used as the main resource for the entire general chemistry sequence at UC Davis. Overall, the ChemWiki has demonstrated its utility to different courses, instructors and teaching styles. Each quarter new material and functions are being added on to the ChemWiki that allow the OER to replace traditional textbooks and adapt to new and innovative teaching strategies.

Conclusions and future directions

The results from our study show that the ChemWiki is a viable alternative to a traditional textbook. With positive results from the use of the ChemWiki in other classes, SS 2014 Chem 2C and FQ 2014 Chem 2A, we believe that the ChemWiki can replace traditional textbooks as the primary resource for the general chemistry courses. The ChemWiki provides a unique flexibility that can adapt to new and innovative instructional techniques. We hope that the data we have provided here, the flexibility, and the significant cost savings of the ChemWiki will allow other instructors to use the ChemWiki as the primary resource in their general chemistry class.

Further, the ChemWiki provides a unique opportunity to study student study habits. Using the traffic on the ChemWiki and the Time on Task survey, we can begin to investigate the type of activity and study habits associated with student performance. We are also developing and integrating other Wikitexts in the STEM Hyperlibrary (including more advanced classes than general chemistry) to provide students a broader picture of how chemistry relates to other core subjects.

Acknowledgements

A preliminary overview of these results was previously reported within an Educause Learning Initiative (ELI) brief (Feb. 2015). The authors would like to thank the many faculty, teaching assistants, and students that contributed to the ChemWiki or helped in the implementation of the assessment pilot; in particular the UCD Chem 2C students participating in the pilot. We would also like to thank the staff in the Chemistry Department and the iAMSTEM hub in Undergraduate Education at UC Davis for providing logistical and analytical support. Support for this pilot came from the National Science Foundation (DUE TUES-1246120) and software and hosting provided by MindTouch.

References

1. Adams W. K., Perkins K. K., Podolefsky N. S., Dubson M., Finkelstein N. D. and Wieman C. E., (2006), New instrument for measuring student beliefs about physics and learning physics: the Colorado Learning Attitudes about Science Survey, Phys. Rev. ST Phys. Educ. Res., 2, 010101.
2. Allen I. E. and Seaman C. A., (2007), Superiority, Equivalence and Non-Inferiority, Qual. Prog., 40, 52–54.
3. Allen I. E. and Seaman J., (2014), Opening the Curriculum: Open Educational Resources in U.S Higher Education, Pearson.
4. Allen G., Guzman-Alvarez A., Molinaro M. and Larsen D. S., (2015), Assessing the Impact and Efficacy of the Open-Access ChemWiki Textbook Project, Educause Learning Initiative Brief.
5. Bacow L. S., Bowen W. G., Guthrie K. M., Lack K. A. and Long M. P., (2012), Barriers to Adoption of Online Learning Systems in U.S. Higher Education, Ithaka S + R.
6. Barbera J., Adams W. K., Wieman C. E. and Perkins K. K., (2008), Modifying and validating the Colorado Learning Attitudes about Science Survey for use in chemistry, J. Chem. Educ., 85, 1435-1439.
7. Baum S. and Ma J., (2014), Trends in College Pricing, College Board.
8. Bliss T., Robinson J. T., Hilton J. and Wiley D. A., (2013), An OER Coup: College Teacher and Student Perceptions of Open Educational Resources, J. Interact. Media. Educ., 4.
9. Bowen W. G., Chingos M. M., Lack K. A. and Nygren T. I., (2012), Interactive Learning at Public Universities: Evidence from Randomized Trials, Ithaka S + R.
10. Buffa M. G. F., (2006), SweetWiki: Semantic web enabled technologies in Wiki, Denmark.
11. Eberlein T., Kampmeier J., Minderhout V., Moog R. S., Platt T., Varma-Nelson P. and White H. B., (2008), Pedagogies of engagement in science, Biochem. Mol. Biol. Educ., 36, 262–273.
12. Hake R. R., (1998), Interactive-engagement versus traditional methods: a six-thousand-student survey of mechanics test data for introductory physics courses, Am. J. Phys., 66, 64–74.
13. Higher Education Opportunity Act (2008), Public Law 110-315.
14. Hilton J. and Laman C., (2012), One college's use of an open psychology textbook, J. Open Dist. e-Learn, 27, 265–272.
15. Hilton J. L., Gaudet D., Clark P., Robinson J. and Wiley D., (2013), The Adoption of Open Educational Resources by One Community College Math Department, The International Review of Research in Open and Distributed Learning, 14.
16. Lindshield B. L. and Adhikari K., (2013), Online and Campus College Students Like Using an Open Education Resource Instead of a Traditional Textbook, J. Online Learn. Teach., 9, 26–38.
17. Lovett M., Meyer O. and Thille C., (2008), The Open Learning Initiative: Measuring the Effectiveness of the OLI Statistics Course in Accelerating Student Learning, J. Interact. Media. Educ., 1, 13.
18. Office G. A., (2005), College Textbooks: Enhanced Offerings Appear to Drive Recent Price Increases.
19. O'Neill M., (2005), Automated use of a wiki for collaborative lecture notes, ACM SIGSCE Bulletin, 37, 267–271.
20. Parker K. R., Parker, K. R., Chao, J. T., Chao, J. T., Green, B. and Green, B., (2007), Wiki as a Teaching Tool, Interdisciplinary Journal of Knowledge and Learning Objects, 3, 57–72.
21. Pence L. E. and Pence H. E., (2015), Using Wikis To Develop Collaborative Communities in an Environmental Chemistry Course, J. Chem. Educ., 92, 86–89.
22. Perry M. J., (2012), The college textbook bubble and how the “open educational resources” movement is going up against the textbook cartel, American Enterprise Institute (accessed February 18th, 2015).
23. Rasmussen A., Lewis M. and White J., (2013), The application of wiki technology in medical education, Med. Teach., 35, 109–114.
24. Robinson J., Fischer L., Wiley D. and Hilton J., (2014), The Impact of Open Textbooks on Secondary Science Learning Outcomes, Educ. Res., 26, 39–49.
25. Rusay R. J., McCombs M. R., Barkovich M. J. and Larsen D. S., (2011), Enhancing Undergraduate Chemistry Education with the Online Dynamic ChemWiki Resource, J. Chem. Educ., 88, 840.
26. Shorb J. M. and Moore J. W., (2010), Enhancing Learning with Online Resources, Social Networking, and Digital Libraries, American Chemical Society, vol. 1060, pp. 283–308.
27. Simkins S. and Maier M., (2010), Just-in-time teaching: across the disciplines, across the academy, Stylus Publishing, LLC.
28. Tsai W. T., Li W., Elston J. and Chen Y. N., (2011), Collaborative Learning Using Wiki Web Sites for Computer Science Undergraduate Education: A Case Study, IEEE Trans. Educ., 54, 114–124.
29. Walker E. and Nowacki A. S., (2011), Understanding Equivalence and Noninferiority Testing, J. Gen. Intern. Med., 26, 192–196.
30. Yang S. C., (1996), Designing Instructional Application Using Constructive Hypermedia, Educ. Tech., 36, 45–50.

---