Click here for this document in PDF format

Evaluating the My Reading Coach program

Sample and Procedure

Longitudinal data from 164 participants who received lessons in the My Reading Coach program were collected between 2001 and 2003. Participants' reading ability was typically tested at the beginning and end of two consecutive academic years, except for 36 participants who were tested during January 2001, May 2001, September 2001, and May 2002 and 29 participants who were tested during September 2002 and May 2003. Although four total assessments were possible for a majority of the sample, not all participants completed the four possible assessments. Each participant's reading score was evaluated an average of three times, yielding 490 datapoints available for analysis. The number of lessons the participants received each year was recorded at the end of the year, but in order to examine the effectiveness of the lessons across all available datapoints, each participant was assigned a value of zero lessons at the beginning of each year/term. Most of the participants were in the second grade (65.85%, n = 108), and the remainder (34.15%, n = 56) was in the first grade when data collection began. Most of the participants (56.1%, n = 92) were enrolled in regular education, 9.15% (n = 15) were classified as learning disabled, 16.46% (n = 27) were ESL students, and 18.29% (n = 30) were in bilingual education programs.

Plan of analysis

In order to examine changes in reading scores over time, the relationship between number of sessions and reading scores, and the effects of group membership (i.e., learning disabled, ESL, bilingual, or regular) on initial reading levels and the change over time in reading levels, multilevel modeling statistical techniques were implemented. These techniques tested for group membership differences in within-person relationships (i.e., an individual's change over time in reading scores). A more detailed description of multilevel modeling and the results of the analyses are presented below.

In the multilevel modeling framework, individual change is represented through a 2-level hierarchical model (Hawkins, Guo, Hill, Battin-Pearson, & Abbott, 2001). The simple form of a multilevel model can be conceived of as two separate models, one a within-person model (Level 1) and the other a between-person model (Level 2). Between-person variability refers to the extent to which people differ from each other, while within-person variability refers to how much people vary from themselves (i.e., fluctuate over time). The multilevel modeling method allows for the simultaneous estimation of both:

1. a separate within-person model of regression slope and intercept for each respondent; and
2. a between-person model in which the within-person slopes and intercepts are treated as dependent variables regressed on person-level predictor variables.

Specifically, multiple observations are seen as nested within the person, and this treatment of multiple observations allows the investigator to proceed without difficulty when the number and spacing of time points varies across persons (Raudenbush & Bryk, 2002). Therefore, this technique allows for the assessment of group differences in students' change in reading scores over time, which provides a basis for making conclusions about the effectiveness of the reading program for students who are in regular education, those classified as learning disabled, ESL, or bilingual.

An important feature of change analysis in multilevel modeling is the assumption that the growth parameters vary across individuals (Raudenbush & Bryk, 2002). A statistical approach that includes time as a component in the model is considered a type of growth curve (trajectory) model (Chou, Bentler, & Pentz, 1998). Specifically, in the Level 1 equation, β_1i is the growth rate for person i over the data-collection period and represents the expected change during a fixed unit of time. The intercept parameter, β_0i, is person i's initial status score. Both the intercept and growth-rate parameter are allowed to vary at Level 2 as a function of measured person characteristics. By examining the shape and rate of change in outcomes across multiple waves of data, multilevel modeling is a powerful and flexible approach for developmental scientists over techniques that treat change in two-wave segments (Schulenberg & Maggs, 2001). As mentioned earlier, multilevel modeling does not assume that data sets are perfectly balanced. Instead, it uses all available data from each participant to estimate a trajectory for that participant, controlling for the timing of that individual's measurements (Karney & Bradbury, 1997). Therefore, even though not all students completed four assessments and the spacing between their assessments was unequal, data available from every participant is included in these analyses.

An example of a model used to analyze the reading program can be expressed as:

Level 1:

READING SCOREit = β0it + β1it(TIME)it + β2it(# OF LESSONS)it + rit

(1)

where READING SCORE_it is the score of Person_i at Time_t, TIME_it indicates the number of months elapsed in the program, # OF LESSONS_it indicates the number of sessions the student completed during each term/year, β_0i is the intercept indicating Person_i's reading score at the first timepoint, β_1i is the slope indicating change in Person_i's reading scores over time, and r_it is the random component or error associated with the scores of Person_i at Time_t. In order to estimate average effects for the entire sample, the intercepts and slopes of the Level 1 within-person model become the outcomes for the Level 2 between-person equations as follows.

Level 2:	β0i = γ00 + γ01(GROUP) + μ0i	(2)
	β1i = γ10 + γ11(GROUP) + μ1i	(3)
	β2i = γ20 + γ21(GROUP) + μ2i	(4)

Equation 2 shows that Person_i's average reading score across the timepoints (β_0i) is a function of the intercept (grand mean) for the entire sample, the effect of his or her group membership (i.e., learning disabled, ESL, bilingual, or "regular"), and a random component or error (&mu_0i). Likewise, equation 3 shows that Person_i's change over time (β_1i) is a function of the grand mean of the entire sample, the effect of his or her group membership, and a random component or error (μ_1i). Lastly, equation 4 shows that Person_i's relationship between the number of lessons completed per term/year and reading scores is a function of the grand mean of the entire sample, the effect of his or her group membership, and a random component or error (μ_2i).

Results

Several researchers (e.g., Nezlek, 2001; Raudenbush & Bryk, 2002) recommend conducting a preliminary analysis to ensure that there is enough variability at Level 1 and Level 2 to warrant continuation with analyses. This preliminary analysis is termed a fully unconditional model (also referred to as a null model), in which no term other than the intercept is included at any level (Curran, 2000; Nezlek, 2001) and is useful because it is the first step in establishing a trend of reading scores improving over time. The fully unconditional model that was used to examine the between-person and within-person variability in reading scores was:

Level 1	READING SCOREit = β0it + rit
Level 2	β0i = γ00 + μ0i

In addition to providing a point estimate and confidence interval for the grand mean, γ₀₀ (Raudenbush & Bryk, 2002), this model also provides information about the outcome σ² variability at each of the two levels. The s2 parameter represents the within-group variability, and τ₀₀ captures the between-group variability (Raudenbush & Bryk, 2002). From these two parameters, the intraclass correlation coefficient was calculated, which measures the proportion of the variance in the outcome variable that is between people (Level 2 units) (Raudenbush & Bryk, 2002). This correlation can be calculated through the following formula:

Results from this analysis indicated that τ₀₀ was 19.42 and σ² was 61.39, resulting in an intraclass correlation coefficient of 0.24. This coefficient is interpreted in terms of variability, such that 24% of the variability in the reading scores was between people and 76% was within people. In other words, participants consistently differed from themselves more than they differed from each other in terms of their reading scores. This finding is not surprising given the longitudinal nature of the data and the expected increase over time in individual's reading scores. Because the intraclass correlation indicated that variability in the outcome variable stems from both levels, the next step of analysis was full model specification in order to attempt to explain some of the within- and between-person variability.

Overall, participants' reading scores significantly improved over time (γ₀₁ = .76, p < .001) and more lessons were positively associated reading scores (γ₁₀ = .09, p < .001). In other words, when a student (regardless of group membership) completed more lessons, his or her reading performance was better compared to times when he or she completed fewer lessons. However, it is also important to consider the contribution of group membership. Results indicate that there were group differences in overall reading scores, in the change over time of reading scores, and in the relationship between the number of lessons and reading scores. Specifically, students enrolled in regular education program had higher reading scores than learning disabled and ESL students. As can be seen from Figure 1, students enrolled in regular education programs seem to be experiencing the most growth in reading scores over time (irrespective of the number of lessons), and it should be noted that the slopes of improvement for the three other groups are significantly different from the slope for regular education (i.e., they are improving significantly less).

Although regular students seem to be improving the most rapidly over time, Figure 2 shows that learning disabled and bilingual students seem to be benefiting the most from more lessons. Specifically, the slopes of learning disabled and bilingual students are significantly different from the slope of students enrolled in regular education. It is particularly interesting to note that bilingual students' reading scores are not significantly different from regular students' reading scores after both groups received 40 lessons. Additionally, the scores of learning disabled students are not significantly different from ESL students' reading scores after 40 lessons.

An additional analysis was conducted to examine the potential acceleration of reading scores as the number of lessons increased. Results indicate that there is a significant and positive acceleration in reading scores (γ₀₃ = .002, p < .05), even when controlling for the effect of time. Figure 3 depicts the findings from this analysis. The potential acceleration of reading scores over time was also examined, but the results of this analysis were not significant. Therefore, the change over time in reading scores appears to be linear.

Discussion

Although the effectiveness of the My Reading Coach program was not tested against a control group, results from the present study yielded several interesting and significant findings. Students in regular education had higher reading scores than learning disabled and ESL students, but students from all groups had improved reading scores with more lessons and their reading scores improved over time. Students enrolled in regular education programs seem to be improving significantly faster than students in the other groups, but learning disabled and bilingual students seem to be benefiting the most from more lessons. Therefore, one way to potentially bridge the gap in reading performance between regular students and those who are learning disabled or bilingual would be to increase the number of lessons available to learning disabled and bilingual students, especially when considering the acceleration in performance when exposed to more lessons.

 

References

Chou, C.P., Bentler, P.M., & Pentz, M.A. (1998). Comparisons of two statistical approaches to study growth curves: The multilevel model and the latent curve analysis. Structural Equation Modeling, 5, 247-266.

Curran, P.J. (2000). A latent curve framework for the study of developmental trajectories in adolescent substance use. In J.S. Rose, L.Chassin, C.C. Presson, & S.J. Sherman (Eds.), Multivariate applications in substance use research: New methods for new questions (pp. 1-42). Mahwah, NJ: Erlbaum.

Hawkins, J.D., Guo, J., Hill, K.G., Battin-Pearson, S., & Abbott, R.D. (2001). Long-term effects of the Seattle social development intervention on school bonding trajectories. Applied Developmental Science, 5, 225-236.

Karney, B. R., & Bradbury, T. N. (1997). Neuroticism, marital interaction, and the trajectory of marital satisfaction. Journal of Personality and Social Psychology, 72, 1075-1092.

Nezlek, J.B. (2001). Multilevel random coefficient analyses of event- and interval-contingent data in social and personality psychology research. Personality and Social Psychology Bulletin, 27, 771-785.

Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods (second edition). Thousand Oaks, CA: Sage.

Schulenberg, J., & Maggs, J.L. (2001). Moving targets: Modeling developmental trajectories of adolescent alcohol misuse, individual and peer risk factors and intervention effects. Applied Developmental Science, 5, 237-253.