|
Click here for this document in PDF format
An Investigation of the Effectiveness of Using My Reading Coach to Improve 2nd Graders' Reading Comprehension
Prepared by: Janna M. Crews
August 26, 2003
An Investigation of the Effectiveness of Using My Reading Coach to Improve 2nd Graders' Reading Comprehension
The main purpose of this study was to determine if the use of My Reading Coach (MRC), a phonics-based, computer-assisted reading tutorial, improves reading comprehension of 2nd grade students. The students participating in the study attend two Title I elementary schools in the Sunnyside School District. One school used the MRC program for 2nd grade reading, while the other school used the in-class reading program Success For All (SFA). This report will refer to these two schools as the MRC school/students and the SFA school/students, respectively. Reading comprehension was measured by pretests and posttests of the Degrees of Reading Power (DRP) exam. This preliminary report outlines the results found. Two analyses were undertaken. First, the pretest DRP unit scores were analyzed to determine if the groups were equivalent. Results indicate there were distinct differences. Second, a repeated measures analysis of the differences from the pretest to posttest scores by treatment (MRC vs. SFA) and type of student (Regular Ed vs English Language Learner) was conducted to investigate the benefits of the different reading programs for different types of students. Posthoc analyses were conducted to further investigate interaction effects and tests for simple effects.
Results indicate that the English Language Learner (ELL) students using MRC improved their reading comprehension significantly more than the ELL students using SFA. In addition, both the MRC and SFA RegEd students achieved statistically equivalent and significant improvement in their reading comprehension. In other words, the MRC ELL students improved more than the SFA ELL students, while the MRC RegEd and SFA RegEd students did equally well. These and other preliminary findings are discussed in further detail below.
Analysis 1
By analyzing the students' pretest scores on the DRP, we can compare the groups within and between the schools at the onset of the study. This analysis resulted in the following findings.
- There is a statistically significant difference between the students in the SFA and the MRC treatment groups at the beginning of the school year. The 2nd grade SFA students (the control group) scored significantly higher on the DRP exam (mean DRP units=17.29) than the 2nd grade MRC students (the treatment group) scored (mean DRP units=4.47), prior to receiving instruction.
- In addition, the Regular Ed (RegEd) versus English Language Learner (ELL) composition of the student population at each school is approximately reversed. The SFA school had 28 RegEd and only 10 ELL students, whereas the MRC school had 13 RegEd and 21 ELL students.
- The RegEd students scored significantly higher on the Pretest than the ELL students at the MRC school (mean DRP units of 13.62 and -1.19, respectively, p<.05). In contrast, there was not a significant difference between the average pretest DRP unit scores of the Regular Ed and ELL students at the SFA school (17.107 vs. 17.8, respectively).
- The ELL students in the SFA group scored significantly higher on the Pretest DRP (mean=17.8), than the ELL students in the MRC group (mean= -1.19, p<.02). However, the difference between the Regular Ed students at the SFA school (mean=17.11) and the MRC school (mean=13.62) was not statistically significant.
Overall, this indicates that prior to any treatment, the students at SFA school, in particular the ELL students, scored statistically significantly higher on the DRP exam than their counterparts at the MRC. In other words, on average the students in the treatment group had lower levels of reading comprehension at the start of the school year (before using MRC) than the students in the comparison group (before using SFA).
Analysis 2
The purpose of this analysis was to compare the effectiveness of MRC in helping 2nd graders improve their reading comprehension (DRP unit scores) as compared to a control group of 2nd graders using SFA. A repeated measures analysis was conducted to investigate the pretest to posttest differences on the DRP exam by treatment (MRC vs. SFA) and type of student (Regular Ed vs ELL). In a repeated measures design, a variable is measured two or more times for each participant, thus requiring fewer participants. In this case, the variable "DRP score" is measured twice; the pretest is the first measure and the posttest is the second measure. Furthermore, because the same participant is repeatedly measured, the repeated measures analysis has the distinct advantage of removing the variance from the error term that results from variability due to individual differences between participants not related to the treatment (Norusis, 1990; Shavelson, 1988). Removing this portion from the error variance increases the power of the statistical test for significance.
The between-subject main effects for treatment and "type of student" are not meaningful in this study. The effects of primary interest are the pretest-postest within-subject effects and any interactions therein. This analysis resulted in the following findings.
- As expected, the mean posttest DRP (mean = 21.569) scores across all groups (treatment and type of student) are significantly greater (p<.007) than the pretest DRP scores (mean= 11.236), indicating that on average the students in all groups improved their reading comprehension during the school year.
- Neither of the within-subject main effects for the repeated measures were statistically significant. In other words, the pretest-posttest differences between the treatment groups (MRC vs. SFA) across both types of students were not significant. Nor were the pretest-posttest differences between the type of student groups (RegEd vs. ELL) across both treatments significant. Thus, although the MRC group scored significantly lower on the DRP pretest than the SFA group, the improvement achieved by the two groups was not significantly differently.
- The within-subject interaction of treatment by type of student by pretest-posttest differences was statistically significant (p<.008). As a result, posthoc analyses were conducted to test for simple effects. The posthoc analysis revealed that the MRC ELL students improved their reading comprehension (post DRP score - pre DRP score) by a significantly greater amount than the SFA ELL students. In fact, while the MRC ELL students improved their scores an average of 18.286 DRP units, the SFA ELL student scores declined an average of 3.4 units. This indicates that the MRC program helped ELL students improve their reading comprehension significantly more than the SFA program. In contrast, the improvements by the MRC RegEd students (mean=3.62) and the SFA RegEd students (mean= 12.39) were not statistically significantly different. These results indicate that the MRC program is beneficial to both RegEd and ELL students, but it is particularly beneficial to the ELL students. This is further illustrated by the average DRP scores of the MRC students. The pretest scores for the MRC RegEd and ELL students are 13.62 vs -1.19, whereas the posttest scores are 17.23 vs 17.10. It appears that although there was a large discrepancy in pretest scores for the two types of students, after using the MRC program the ELL students scored, as well as the RegEd students.
Limitations
This study has an unbalanced design with small sample sizes in two cells (MRC RegEd = 13 students, SFA ELL = 10). Furthermore, the Regular Ed (RegEd) versus ELL composition of the student populations at each school is approximately reversed. This design reduces the power of the statistical tests and increases the probability of Type I errors - a false negative result.
In addition, the pretest DRP scores of the students in the two treatment groups were significantly different, indicating there may be further differences between the populations.
Future studies should try to achieve a more balanced design (i.e., equal numbers of students in the different groups) and as many students as possible.
