Title: Relative effects of CMI and non-CMI practice quizzes upon subsequent test performance
Author(s): Blumenfeld, Gerald J.; Darr Jr., Ralph F.
Source: Journal of Research on Computing in Education, Summer93
Vol. 25 Issue 4, p431, 8p, 4 charts
Intended Audience: Educators, school managers and others interested in the effects of computer-managed instruction programs on test performance.
Copyright:
Article Topic: The researchers investigated the effects of computer-based instruction at the University of Akron. They were able to statistically analyze the results of two groups of students taking the same tests, one group using computer-managed instruction (CMI) practice quizzes and one group using non-CMI practice quizzes. The null hypothesis was that the students using CMI practice quizzes would perform better on the tests.
Summary
Researchers at the University of Akron were given an opportunity to investigate the assumption that the use of computer managed instruction practice quizzes improved test results when the University's Center for Computer Based Education encountered problems with their CMI program and had to shut down the use of the CMI program. Throughout the year, students did not have access to the CMI practice quizzes and the researchers were able to use them as a control group.
The CMI program had provided a system of checks and measures, based on Skinner's approach to learning machines. Students were required to take practice quizzes after covering small amounts of information. The program provided immediate feedback on the correct and incorrect answers. If a student moved onto the next quiz for the next unit after successfully answering at least 10 of 12 questions. If a student scored less than 10, the program would generate 12 new, different questions based in the same unit after a short delay. Students were encouraged to work at their own pace and lectures related to the units were held to support the material. Comprehensive exams were taken after successful completion of the practice quizzes.
When the computer program was shut down for the school year, students were required to take the practice quizzes using pencil and paper. Students were given a scoring key and used the quizzes on their own time. The instructor could not monitor the non-CMI students use of the practice quizzes, the student could not take more than one practice test and the computer feedback was absent with the non-CMI students. In all other ways the course remained the same: same tests, quizzes, schedules, and criterion.
The researchers intended to study the affect of the computer-managed instruction on test performance and the hypothesis was that there would be a difference in student test performance between students learning with CMI and those learning without CMI.
Conclusions drawn by the author
The researchers reported that there were no significant differences with test results from the students completing the course using CMI during the three semesters prior to the study of the non-CMI group. This was concluded after using chi-square and comparing proportions of students passing and failing. The raw score for passing this course was set at 23 or greater, with a score of 22 or less resulting in a no-pass condition.
It was also concluded, from statistical analysis, that students attaining no-pass scores increased as they moved from early units to later units. This number also increased at a greater proportion with non-CMI students.
The difference between the scores for the CMI group and the non-CMI group were shown to be significant at or beyond the .01 level. The researchers concluded from this that the CMI students' performance was significantly different from the non-CMI performance.
The researchers concluded that the CMI student receives an intensive instructional experience and benefits more from this experience than the instructional experience of the non-CMI student. This is attributed to the computer response being contingent upon the student's response, and the assumption and observation that multiple opportunities to take practice quizzes with diagnostic feedback facilitates student achievement more so than single paper and pencil tests.
Are the conclusions warranted from the information provided?
The researchers were able to utilize statistical information to test a number of their assumptions and hypothesis. Based on the statistical analysis provided, the researchers conclusions were valid.
Possible unstated beliefs or assumptions
In this study, the researchers presented a number of assumptions that could have been made and not stated. These assumptions were investigated and tested using available data. They discussed alternative explanations based on possible assumptions and then proved the results were still valid using data from the university.
Two different professors taught the courses during the three CMI semesters. However, they overlapped and taught the course concurrently during one of the semesters. The same teaching assistant worked with both professors. Therefore, the three semesters were tested and there were no significant differences found in significance levels of .05 or .01. It was concluded that instructor differences did not affect the data.
It was also assumed, but not stated, that the academic abilities of the students in each group were equal. The researchers did a test using grade point averages of the students and found no significant differences at the .05 level for any of the groups. This of course assumes that the grade point averages were significant measures of the students' abilities.
A final assumption made was that the non-CMI students actually used the practice quizzes. The students worked on these quizzes during their own time and had the answer keys with them. It was no mentioned if there was any accountability placed on the students to show that they completed the practice quizzes. The CMI students were required to take the computer-managed quizzes in order to move forward and take the comprehensive exam at the end of each unit.
Credible Conclusions?
The conclusions of this study are credible. The statistical analysis shows differences between the two groups at the .05 level of significance. The researchers carefully discuss the possible errors in assumptions and have provided evidence that these assumptions do not alter the conclusions.
The authors present information that is not contrary to personal experience and observations. There is sufficient discussion of the literature review to show precedence for their conclusions, without showing a previous bias to their conclusions.
The researchers admit that there is always chance of error with data obtained from natural event. However, the logical approach to the comparisons and the methods used to test variables such as different professors and possible variances in academic strengths of subjects lends credibility to the studies conclusions.
Possible Further Information
This study was competed in 1986. It would be interesting to attempt to replicate the study to test if the differences are still significant with students who are more used to technology. It would also be informative to see if there would still be significant difference if non-CMI students were provided with other forms of reinforcement and practice. Otherwise, this study was well constructed and the conclusions significant.