• E-mail
• Print
• Comment
• Font Size
• Digg
• del.icio.us
• Discuss article

Science Achievement in Social Contexts: Analysis From National Assessment of Educational Progress

Posted on: Wednesday, 10 November 2004, 03:00 CST

ABSTRACT

Substantial gaps in achievement of groups of individuals of varying social status are well documented, but there are no conclusive explanations for them. The author integrated psychological and educational perspectives to scrutinize the effects of 4 protective factors believed to promote academic resilience of students at risk because of low socioeconomic status, minority status, or gender. The analysis was based on data collected from national samples of students in Grades 4, 8, and 12 as part of the National Assessment of Educational Progress 1996 science main assessment. The hierarchical linear models used in this study provided reliable estimates of the sources of achievement gaps and the conditions that contributed to resilience of individuals with high statistical probabilities of academic risk.

Key words: National Assessment of Educational Progress; science achievement; students at risk due to low socioeconomic status, minority status, and gender

Students at risk is a major topic of education policy and discussion. Many researchers have focused on describing conditions associated with the statistical risk of undesirable outcomes among individuals who are members of groups characterized by problems such as poverty and social disadvantage. When groups of students with similar backgrounds are compared, findings indicate that students from families with high socioeconomic status (SES) outperform students from low-SES families; Asian and White students have higher achievement than do Black or Hispanic students; and boys perform better than do girls (Coleman et al., 1966; Gibbons, 1992; Hilton & Lee, 1988; Hoffer, Rasinski, & Moore, 1995; Madigan, 1997; Mason & Kahle, 1989). No single explanation is sufficient to account for those observed average differences in science achievement. Even more of a mystery is why some resilient students "beat the odds" and demonstrate high achievement despite statistical predictions to the contrary.

One of the most commonly investigated risk factors is low SES. Low achievement is attributed to the paucity of resources available to persons with low income, which results from low levels of parental education, low-status parental occupation, large family size, and absence of one parent (Luthar, 1991). Empirical findings show that risk factors have a reciprocal relationship with one's social class status (Garmezy, Masten, &. Tellegen, 1984; Masten et al., 1988). High SES is associated with greater social support, fewer school and behavior problems, and greater social competence. Reviews of resilience and vulnerability to adverse outcomes of childhood poverty have emphasized the necessity of exploring the means by which such processes occur (Garmezy, 1991; Rutter, 1994).

Theoretical Framework

One of the most controversial findings in studies of adolescent achievement is an association with race and ethnicity. Although there is considerable consensus that racial-ethnic differences in school performance are genuine, there is less agreement as to their causes (Lynn, 1977; Mickelson, 1990; Mordkowitz &. Ginsburg, 1987; Ogbu, 1978, 1991, 1992; Sue &. Okazaki, 1990). Parallel studies on children in highly stressed urban environments indicate that expectations and beliefs operate as protective factors that moderate the resilience or vulnerability of affected individuals (Clausen, 1991; Cowen, Work, &. Wyman, 1992; Israelashvili, 1997; Wyman, Cowen, Work, & Kerley, 1993). Other studies suggest that in science, academic resilience of minority students may be improved when teachers emphasize instructional practices that provide greater access to laboratory experiences (Von seeker, 2002; Von seeker & Lissitz, 1999).

Gender has been implicated as an important influence in explaining resilience (Rutter, 1979; Wemer &. Smith, 1982). The implications for gender differences in achievement are provocative, particularly because they are accompanied by a considerable body of theoretical and empirical evidence that documents the psychological impact of gender differences in self-esteem on normal development (Gore & Eckenrode, 1994). Developmental research studies show that girls' sense of negative emotion is greater than that of hoys and is tied to concerns about relationships (Czikszentmihallyi &. Larson, 1984; Larson & Asmussen, 1991). Rush and Simmons (1987) suggested that early adolescent girls experience higher levels of stress than do hoys as a result of transition to high school because of the greater dependence of their self-esteem on expectations of others and because of their need to establish personal social ties, (lender- related differences in self-esteem could influence the beliefs of girls about their ability to do well in science and their willingness to take advantage of educational opportunities.

Although isolated risk indicators such as SES, racial-ethnic status, and gender may be highly predictive, they should not he interpreted as conclusive. Risk is the heightened probability of an undesirable outcome for a population, not for an individual (Render &. Losel, 1997; Garmezy & Masten, 1986). That subtle but significant distinction means that individuals are not at risk for low science achievement because they are poor, female, or a minority; rather, they are part of highly variable risk populations. Overlooking the distinction ignores substantial within-group heterogeneity that accounts for most of the variance in science achievement. Characterizing individuals on the basis of group achievement fails to reveal the qualities or factors that predispose resilient individuals to overcome the stereotypes and stigmas predicted by group affiliation (Catterall, 1998; Nettles & Meek, 1994; Richters &. Weintraub, 1990). Theoretical guidance for investigations about why, on average, groups of students consistently perform more poorly in science, and why some individuals in these groups beat the odds and perform well, is found in a strand of research in developmental psychology and psychopathology that deals with risk and resilience.

Garmezy and colleagues (1984) defined psychological resilience as the manifestation of competence despite exposure to risk mechanisms that lead to deleterious outcomes. Resilience is not a rare occurrence. As many as one fourth to one half of children exposed to severe stress and adversities develop into competent and caring adults and do not succumb to psychopathology (Reynolds, 1998; Rutter, 1985; Werner, 1989). Growing awareness of the ubiquity of resilience is coupled with the recognition that simple models of risk are inadequate to explain how the interplay of complex developmental processes and protective mechanisms with risk factors foster resilience in social contexts (Masten et al., 1988; Rutter, 1987; Seifer & Sameroff, 1987). There is ample theoretical and empirical evidence that demographic differences should he conceptualized as social contexts, or collections of variables, that alter the psychological significance and social demands of life events and affect subsequent relationships between risk and resilience.

Garmezy (1983, 1985) identified three broad sets of protective factors that contribute to resilience and moderate predictions of vulnerability according to an individual's risk status: (a) personality attributes such as attitudes and beliefs, (b) quality of home environments, and (c) quality of external support systems such as schools. Substantial empirical evidence indicates that resilience of children is patterned, at least in part, by those variables (Rrooks, 1994; Ftoyd, 1996; Garmezy, 1983, 1991; Gonzalez & Padilla, 1997; Luthar & Zitier, 1991; Rak & Patterson, 1996; Rutter, 1987). Although Garmezy did not focus specifically on educational outcomes, his assessment resonates with results from educational research (Gatterall, 1998; Goleman & Hofier, 1987; Freiberg, 199?; Gonzalez & PadiUa; Lee, Ghen, & Smerdon, 1996; Peng & Wright, 1994; Rak & Patterson; van Welzenis, 1997; Wang & Gordon, 1994). Such evidence provides a starting place for one to understand how students' attitudes and beliefs, home learning environments, and instructional opportunities can foster academic resilience.

Protective Effects of Positive Attitudes and Beliefs

Some educators argue that fostering positive attitudes about science is one of the most desirable outcomes of science education (Garey & Shavelson, 1988; George & Kaplan, 1998; Greenfield, 1996; Raizen & Jones, 1985). Researchers have long believed that positive attitudes increase formal and informal science learning after the direct influence of the teacher has ended (Mager, 1968). Recent studies provide evidence that positive attitudes foster science achievement and career interest by increasing the likelihood that students will enroll in advanced science courses and engage in future activities associated with lifelong science learning (Garey &. Shavelson; Mason &. Kahle, 1989; Norwich & Duncan, 1990).

Published results from the National Assessment of Educational Progress (NARP) 1996 science assessment (O'Sullivan & Weiss, 1999) show that attitudes toward sc\ience vary significantly for hoys and girls and for members of different racial-ethnic groups, particularly by 12th grade. Women and minorities, two groups at high risk for low achievement, are more likely to have negative attitudes ahout the significance of science in everyday life and poor self- concepts regarding their ability to do well in science. Research suggests that the relatively poor academic performance of female and minority students stems, at least in part, from their more negative attitudes and beliefs ahout science (Clewell, Anderson, & Thorpe, 1992; Skolnick, Langbort, & Day, 1982).

Protective Effects of Opportunities at Home

One of the oldest and most persistent explanations for achievement differences is that educational opportunities provided at home affect students' readiness to learn (Coleman et al., 1966). Whereas hundreds of studies attest to associations between home environments and achievement throughout a child's school career, less attention has been given to significant systematic variability in achievement among students from families with access to similar home environments. What is not clear is how the level of parent education and access to a rich collection of literacy-based materials in the home interact with risk factors to influence learning readiness and subsequent achievement. I examined the extent to which home-learning opportunities matter for students at different grade levels, and whether, on average, students with different statistical probabilities of risk benefit equally from these opportunities.

Protective Effects of Opportunities at School

The National Science Education Standards (National Research Council, 1996) include specific guidelines that describe science instruction for elementary and secondary classrooms. The standards prescribe science-learning experiences in which students have ample opportunities to conduct hands-on inquiry in their classrooms. Substantial empirical and theoretical evidence indicates that those kinds of learning experiences are a starting point for personal construction of meaning and can lead to higher achievement for all students (Anderson, 1998; Burkam, Lee, & Smerdon, 1997; Carey, 1985; Carmichael et al., 1990; Ertepinar &. Geban, 1996; Freedman, 1997; Glasson, 1989; Lee, Chen, & Smerdon, 1996; Odubunmi & Belogun, 1991; Piaget, 1970; Stohr-Hunt, 1996; von Glaserfeld, 1984, 1987; Von seeker &. Lissitz, 1999). One question that has not been well explored is whether the "payoff" associated with specific instructional practices varies significantly for individuals with different probabilities of academic success.

One of the criticisms of the science education standards is that, although they emphasize the importance of promoting science achievement for all students regardless of demographic status, proposed science education reforms do not directly address theoretical issues surrounding ethnic, socioeconomic, and gender equity (Rodriguez, 1997). Although some evidence shows that instructional emphases explain discrepancies in student achievement (McCauley, 1995; National Center for Education Statistics, 1992), there is no empirical evidence that supports the viability and utility of proposed instructional reforms for creating more equitable opportunities nationwide (Donmoyer, 1995; Richard, 1994). In general, evidence that explains how learning experiences interact with combinations of student background variables for influencing science achievement is sparse.

Most of the research on psychosocial risk factors has provided data on how to identify risk variables. However, those epidemiological models are limited in that outcomes are easily attributed to risk-related or compensatory factors even though the nature and extent of the mechanisms that account for their influence remain unknown (Compas, Howell, Phares, Williams, & Giunta, 1989; Masten &. Garmezy, 1985; Richters & Weintraub, 1990; Rutter, 1994). Over the last 15 years, the research agenda in developmental psychology has embraced models that focus on how protective effects interact with social context to moderate the influence of risk factors (Blocker & Copeland, 1994; Floyd, 1996; Garmezy, 1991; Grossman et al., 1992; Israelashvili, 1997; Jackson, Bom, & Jacob, 1997; Luthar, 1991; Radke-Yarrow & Sherman, 1990; Rutter, 1987; Werner, 1989; Werner & Smith, 1992). An emerging focus of that research strand has been on resilient individuals who defy expectations by developing normally and coping with their lives effectively. I extend that focus by examining the impacts of four protective factors on students with varying probabilities of academic risk. The research questions were as follows:

1. To what extent do differences in the associations of protective factors with science achievement vary across Grades 4, 8, and 12?

2. To what extent do psychological and environmental protcctive factors influence academic resilience'?

3. On average, are students from families of low SES more likely to demonstrate academic resilience when parent education is high and home environments promote literacy?

4. On average, are minority students more likely than majority students to demonstrate academic resilience when they have positive attitudes about science and have access to laboratory-based instructional opportunities?

5. On average, are boys and girls who like science and are confident about their ability to do well in science equally likely to demonstrate academic resilience?

Investigation of those questions with NAEP survey data cannot produce causal evidence that explains low achievement of students at risk. Nonetheless, recognition of the magnitudes of the effects of protective factors in different social contexts can illuminate understanding of how psychological and environmental mechanisms influence science achievement. Findings can stimulate and inform discussions about why some groups of individuals are more vulnerable statistically than others and why some individuals at risk are more resilient than others.

Method

Estimation of the effects of protective factors on science achievement is complicated by methodological challenges posed by the NAEP data collection design, namely, the multistage cluster sampling and unequal selection probabilities associated with stratification and oversampling of certain subpopulations, and the measurement error associated with the matrix sampling scheme for the outcome variable. If those issues are not handled appropriately, results of policy analyses could be misleading. The hierarchical linear models (HLMs) and HLM software used in this analysis accommodated those considerations.

Sample Selection

Multistage catering. As the Nation's Report Card, NAEP must report accurate results for populations of students and subgroups of these populations (e.g., minority students, students attending nonpublic schools). To ensure accurate results, the relatively small samples of students selected for the NAEP assessments must represent the entire student population. I selected students as part of a two- stage cluster sample (students within schools) with stratification at the first stage. Schools were initially stratified on the basis of urbanicity, minority concentration, size, and area income, and then schools within each stratum were selected at random. Students were selected at random within schools.

Sampie weights. Researchers derive estimates of population and suhpopulation characteristics in NAEP reports by using sample weights. In the 1996 NAEP assessment, deliberate oversampling of certain populations (e.g., private schools and public schools with moderate or high enrollments of Rlack or Hispanic students) enhanced the reliability of estimates for the oversampled subgroups but produced a sample containing proportionately more members of these subgroups than were in the population. In addition, nonresponse on the parts of schools and students resulted in a final sample that was unrepresentative of the number and types of students who would have been found in a target population in 1996. Weights were assigned to each student and school to account for the unequal probabilities of selection and to adjust for nonresponse. Johnson, Qian, Wallace, and Rust (1999) provided a technical description of the weighting procedures used in the 1996 NAEP assessment.

Analytic sample. The samples that I analyzed in this study were 7,288 Grade 4 students, 7,741 Grade 8 students, and 7,516 Grade 12 students from the 1996 NAEP main science assessment reporting sample for whom the following information was available: (a) science achievement data, (b) Title 1 funding status (a measure of SES), (c) racial-ethnic status, and (d) gender. The contribution of each case was weighted according to its probability of selection so that parameter estimates produced by HLM software would be unbiased.

Science Achievement

Matrix sampfmg plan. One of the challenges of working with NAEP data is to provide for the special character of the outcome variables used for assessing achievement. Because of limitations on the amount of test time available and considerations of statistical efficiency, I observed students on only a subset of relevant items. Each student received an assessment booklet that contained (a) a set of general background questions; (b) a set of subject-specific background questions; (c) a set of questions about his or her motivation and familiarity with the assessment materials; and (d) up to two sets, or blocks, of cognitive questions that assessed the knowledge and skills outlined in the subject-area framework. A consequence of using a matrix sampling scheme was that no student had a complete outcome score.

Estimating student outcomes. NAEP researchers used scaling models to summarize students' performance and to account for substantial amounts of missing data. I used multiple imputation procedures to produce five plausible values that estimatedstudents' true proficiency given the pattern of item responses and other characteristics of the students. 1 used HLM software to compute separate analyses for each of the five plausible values of science achievement and produced a typical score for each student in Grade 4, 8, or 12 that took into account the extra uncertainty that arose because multiple plausible values, rather than a single observed outcome, were available (Bryk, Raudenbush, & Congdon, 1996; Mislevy, Johnson, &. Muraki, 1992; Rubin, 1987).

Academic Risk

Three demographic characteristics associated with academic risk were low SES, being a member of a minority racial-ethnic group, and being female.

SES. One measure of SES available in the NAEP data files is student eligibility for Title 1 funding. Students who received Title 1 funding were assigned a value of 1; ineligible students were assigned a value of 0.

Racial-ethnic status. The NAEP data files include information about students' race and ethnicity. Black, Hispanic, and American Indian students were assigned a value of 1 (minority) and White and Asian American students were assigned a value of 0 (majority). The aggregation of race-ethnicity was justified because of the similarities in the distribution of science achievement among groups of students assigned each value and the need to maintain sufficient subgroup size within schools.

Gender. Students' gender is identified in the NAEP dam files. In this study, girls were assigned a value of 1, and hoys were assigned a value of 0.

Protective Factors

The overarching goal of this research was to investigate the impact of four protective factors-level of parental education, home environment, attitudes and beliefs about science, and quality of students' instructional opportunities.

Parental education. One of the factors that researchers believe influences students' achievement is level of education attained by their parents. At each grade level, students were asked to indicate the extent of schooling for each of their parents. The information was combined into one parent education reporting variable through the following process. If a student indicated the extent of education for only one parent, that was included in the data. If the student indicated the extent of education for both parents, the higher of the two levels was included in the data. In this study, students were assigned a value of 0 if their parents did not finish high school, graduated high school, or had some education after high school, and a value of 1 if one or more parents graduated from college.

Home environment. The NAEP dataset includes a number of student background items that reflect the educational environment of the home. One composite variable available in the student file, HOMEEN, is based on student responses to four questions about the availability of a newspaper, encyclopedia, magazines, and more than 25 books. That variable provides the most reliable indicator of the amount of literacy-based materials in the home and is used frequently in NAEP analysis as a predictor of the influence of home environments on achievement.

Attitudes and beliefs about science. The NAEP student questionnaires for Grades 4, 8, and 12 include eight identical items that measure students' attitudes about science and their beliefs about their ability to do well in science. I used principal components analysis to extract a composite factor that captured those attitudes and beliefs.

Learning experiences. Students at each grade answered a common set of questions that provided a profile of the extent to which their instructional experiences in science reflected the ideals set forth in the National Science Education Standards. I used principal components analysis of aggregated student responses to construct a composite factor that measured the extent to which students' teachers in each school emphasized scientific inquiry.

Analytic Model

A pervasive problem in educational research has been the failure of researchers to account for the complexity of the sampling design in analysis of educational outcomes (Bock, 1989; Bryk &. Raudenbush, 1992; Burstein, 1980; Goldstein, 1987; Kreft & De Leeuw, 1998). Because the organizational structure of schools and classrooms is by nature hierarchical, a properly specified model must account for data that arise from a clustered sample. Aggregating individual data to a higher level of the hierarchy (e.g., the school) and analyzing means leads to aggregation bias by ignoring within-group variation that accounts for most of the variability in student achievement. Disaggregation of group data to the individual level (e.g., students) violates the assumption of independence that underlies regression models. Failure to account for dependencies in the data associated with group membership leads to underestimation of the standard errors of model parameters.

HLMs resolve problems of aggregation bias and imprecision inherent in analysis of multilevel data by appropriately adjusting standard errors for cluster effects (Bock, 1989; Bryk & Raudenbush, 1987, 1992; Burstein, 1980; Burstein, Linn, & Capell, 1978; Goldstein, 1987). I used the HLM software to estimate the impact of protective effects in terms of value-added indices that assessed the impact of protective effects after controlling for risk factors associated with low SES, race-ethnicity, and gender. My use of a hierarchical model and value-added approach "leveled the playing field" and allowed meaningful comparisons of risk and resilience among students who were clustered in different schools. A technical description of HLM and its applications is available in Bryk and Raudenbush (1992).

Preiimmary analysis. A fully unconditional, two-level HLM partitioned variance in science achievement into that part that was unique to schools or to individuals. I used the estimates of the variance components to calculate the intraclass correlation (ICC), an index that measured the degree to which the science achievement of students in the same school was more similar than was science achievement of students in different schools. Using HLM to control for duster effects is justified even when ICCs are as low as 0.02 (Kreft &. De Leeuw, 1998). The ICCs for the three grades investigated in this study were .34 (Grade 4), .32 (Grade 8), and .28 (Grade 12). That result indicated that as much as one third of the variability in science achievement in Grade 4, 8, or 12 was caused by differences in the average achievement of students attending different schools. Approximately two thirds of the total variance in achievement could be explained by differences among students. That finding supported the use of a multilevel model of student achievement (HLM) that could explain differences in achievement even after school-level differences were controlled.

The sociai context model. I used three student characteristics- SES, minority status, and gender-to account for differences in achievement that could be explained by some students having a greater probability of academic risk than others. Because preliminary analysis revealed that the interactive effects of the risk factors with each other were not significant, I excluded their two-way and three-way interactions from the social context model.

I included three student-level protective factors-level of parental education, quality of the home environment, and attitude about science-in the model to explore the influence of these effects for all students, regardless of risk. I included four interactions at the student level (SES Parent Education; SES Home Environment; Minority Status Attitude; and Female Attitude) and one cross- level interaction (Instructional Opportunities Minority) to answer questions about the relative contributions of protective effects for students in different social contexts.

Results

In general, results of statistical tests are considered significant if values of p are less than .05. One of the limitations of this interpretation is that the calculation of the value of p for a test statistic (e.g., t tests used in HLM) depends, in part, on sample size and variability. Analysis of a large sample may produce a statistically significant result (i.e., p < .05) that has limited practical value. Conversely, statistical tests conducted on small, highly variable samples can produce p values that are not statistically significant even when the practical effects of a treatment are large. For that reason, the influences of variahles included in the model are presented as effect sizes (ES) rather than as p values.

ES estimates are standardized measures of the significance of statistical tests that allow comparison of outcomes with different metrics and yield results that are less sensitive to sample size and variability. In educational research, effect size values of .10, .30, and .50 SD are interpreted as small, medium, or large, respectively (Cohen, 1988). Effect sizes less than .10 are trivial and of no practical significance even when the p values associated with them are statistically significant.

TABLE 1. Hierarchical Linear Model (HLM) of Risk and Resilience

Associations of Protective Factors With Academic Resilience in Grades 4,8, and 12

The HLMs of risk for Grades 4, 8, and 12 estimated that approximately 65% to 70% of the variability in science achievement was attributed to differences among students who attended the same schools. Less than 14% of the variability at each grade level was explained by risk factors associated with SES, minority status, and gender. However, when protective effects were considered, model fir improved by more than 40% in Grades 4 and 8 and by more than 250% in Grade 12. The social context models at each grade level explained about 20% of the differences in achievement of students who attended the same schools.

Associations of Risk Factors With Academic Resilience

Achievement data presented in Ta\ble 1 show differences in science achievement that were associated with risk status, protective factors, and relevant interactions. In Grades 4, 8, and 12, the science achievement of students from poor families was between two thirds and 1 SD lower than for students from families of average SES or above, regardless of the gender or minority status of those students. Science achievement of minority girls and boys was .72 to more than 1 SD lower than that of majority students, even when SES was the same. Differences in the science achievement of girls versus buys, although statistically significant in all grades, had no practical significance (ES < .1) after socioeconomic and minority status were controlled. The effects are summative. For example, the predicted science achievement of majority students in Grade 8 from more advantaged families was more than 2 SD higher than that of minority students from poor families.

Significant differences in achievement, which could he attributed to the protective effects of personal beliefs and greater academic support at home, occurred even after student demographic status was controlled. On average, students whose parents were college graduates and whose attitudes about science were positive had science achievement scores that were about .5 SD higher than those of students with neither of these benefits. In all grades, the unique contribution of home environments was small after other protective effects were controlled.

Associations of Protective Factors With Academic Resilience in Social Contexts

I expected that protective factors would promote higher science achievement in some social contexts than in others. I also expected that parents' education and home environments would be associated with academic resilience among students from families of low SES. I predicted that minority students would demonstrate higher achievement when they believed that success in science was a realistic outcome and when teachers provided them with opportunities for hands-on inquiry. I also theorized that girls who liked science and anticipated doing well in science would have higher achievement than would girls whose attitudes about science were negative.

TABLE 2. Associations of Parental Education and Home Resources With Science Achievement

Protective factors associated with academic resilience of students of low SES. Luthar (1991) theorized that low levels of parent education and a paucity of resources in students' homes account for low achievement among students from low-SES families. Results provided in Table 1 show that, on average, students in all grades were more likely to demonstrate slightly higher-than- predicted science achievement (ES ≤ .2) when one or more parents had graduated from college or when home environments were more advantaged. However, no evidence indicated that those protective effects were associated more strongly with higher levels of academic resilience among students at risk for low achievement because they were poor. Among students in Grade 12, the small negative interaction of the effects of SES with parents' level of education suggested that by the end of high school the protective effects were smaller for students from poor families than for more advantaged students.

Descriptive analysis of characteristics of students in Grade 4, 8, or 12 in 1996, shown in Table 2, further illuminated results of the inferential statistical analyses conducted with HLM. Science achievement for the least advantaged individuals from families of average or above-average SES (e.g., parents did not graduate from college, literacy-poor home environments) was consistently higher than for the most advantaged individuals from poor families. For example, the average achievement of high school seniors whose parents did not graduate from college but were from families of average or above-average SES was .44 SD higher than that of low-SES high school seniors whose parents had college degrees. Although the science achievement of low-SES 8th-grade students from literacy- rich home environments was .71 SD higher than that of poor students from disadvantaged home environments, their achievement was .70 SD lower than that of comparahle students from families of average SES or above.

For low-SES students, the protective effects associated with level of parent education and literacy-rich home environments were associated more strongly with achievement as students got older. Among low-SES students, the gap in science achievement associated with level of parent education was .37 SD greater in Grade 12 (.42 SD) than in Grade 4 (.05 SD). Likewise, the academic benefits of having home environments that promoted literacy were more evident for students in Grades 8 and 12. Low-SES students without those advantages were the lowest scoring group of students at all grade levels.

The trends suggest that science achievement of low-SES students whose parents do not have a college degree or whose homes have few literacy-based resources, or both, will fall further behind that of their more advantaged peers as these students progress through school. The compensatory effects of the two protective factors (parent education and home environment) are associated with greater academic resilience among students from low-SES families, but they are insufficient to overcome the negative academic consequences of poverty.

Protective factors associated with academic resilience of minority students. One of the questions investigated in this study was whether minority students were more likely than were majority students to demonstrate academic resilience when schools provided instructional opportunities consistent with those recommended by the National Science Education Standards (National Research Council, 1996). Previous studies have demonstrated positive associations of emphasis on scientific inquiry with achievement of minority students. In this study, instructional experiences were not associated with higher minority achievement in Grades 4 and 8, but achievement of Grade 12 minority students was higher (ES = .25) when teachers included additional laboratory opportunities.

TABLE 3. Associations of Science Attitudes and Instructional Opportunities With Science Achievement of Minority Students

There was no evidence that having more positive attitudes about science compensated for risks associated with minority status. On the contrary, the small negative relationship hetwecn minority students' attitudes and science achievement of minority students increased as students got older. A description of the associations of science attitudes and instructional opportunities with science achievement of majority and minority students is shown in Table 3.

Positive attitudes appear to contrihute to academic resilience among minority students hut they do not compensate fully for risks associated with minority status. For example, achievement of minority 4th graders with positive attitudes about science was .35 SD higher than for minority 4th graders whose attitudes about science were negative. However, the average science achievement of minority 4th graders who liked science and were confident ahout their ability to do well was .62 SD lower than that of majority 4th graders with negative attitudes about science. The gaps in science achievement for students with positive versus negative attitudes about science were progressively greater as students got older. When students with the same minority status were compared, the protective effects of attitude for minority students were .13 SD, .40 SD, and .59 SD smaller in 4th, 8th, and 12th grades, respectively.

Science achievement of all students whose teachers provided frequent opportunities for hands-on investigations was higher than that of students whose teachers placed less emphasis on this approach. The influence of instruction on achievement of minority students appears more important when students study more advanced science content, in Grade 4, differences in achievement of minority students were less influenced by teachers' practices (ES < .1) than in Grade 8. By Grade 12, however, science achievement of minority students whose teachers provided frequent opportunities for hands- on investigation was almost 1 SD higher (ES = .98) than that of minority students whose teachers did not. Science instruction matters, but when majority and minority students are in the same class and have equal access to instructional opportunities, the expected achievement of minority students is still inferior, particularly in the lower grades.

Protective factors associated with academic resilience of gifts. One of the explanations proffered to explain the gender gap in science achievement is that girls are less confident in their ability to do well in science. Boys are more likely than girls to have positive attitudes about science, particularly by middle school or high school. For example, the attitudes of 8th-grade students in this study were almost the same. By 8th grade, however, the attitudes of boys were slightly higher (ES = .1) than those of girls- a difference that was sustained through 12th grade. Yet, the results of the hierarchical analysis demonstrate that even when the attitudes of boys and girls about science were the same, the protective effect was slightly smaller for girls than for boys (ES < .10), regardless of grade level.

This study produced no empirical evidence to support theoretical beliefs that gender-related differences in self-esteem influence girls' attitudes about their ability to do well in science or their willingness to take advantage of educational opportunities. Table 4 illustrates that gender gaps in achievement persist and widen over time even when girls' attitudes are as positive as those of their male counterparts. Among 4th-grade children, science achievement of students with positive attitudes was about .5 SD higher than for s\tudents with negative attitudes, regardless of gender. The difference for girls was relatively stable; between 4th and 12th grades, there was less than .1 SD difference in science achievement of girls who had positive attitudes versus girls with negative attitudes about science. However, the gap in achievement between boys with opposite attitudes increased from .51 SD in 4th grade to .61 SD in 8th grade, and .88 SD in 12th grade.

TABLE 4. Associations of Science Attitudes With Science Achievement of Boys and Girls

Discussion

Most of the variability in achievement has long been attributed to what children bring to school in terms of their demographic status, home environment, and readiness to learn (Coleman et al., 1966). With the HLM analysis conducted in this study, I found that approximately 65% to 70% of the variability in achievement could be explained by differences among students who attended the same schools. Three demographic measures of academic risk-ES, minority status, and gender-accounted for 14% or less of that variability. Social contexts models that included four protective effects and their interactions with risk factors improved the model fit by 40% in 4th and 8th grades and by more than 250% in 12th grade. The results suggest reasons that some groups of individuals statistically are more vulnerable than others, as well as the challenges faced by even the most resilient at-risk students.

Education reform is grounded in a belief that understanding the impact of even relatively small protective effects is essential for student success. The results presented in this study justify two conclusions about risk and resilience, namely that (a) personal attitudes and beliefs about what is possible-parent education, home environments, and instructional opportunities-can compensate somewhat for risks associated with low SES, minority status, or gender; but (b) disadvantaged students get less "bang for the buck" from the benefits attributed to these factors.

Low achievement among students who live in poverty is attributed, in part, to lack of academic and social support that results from low levels of parent education and access to fewer learning resources at home. On average, science achievement of all students in Grades 4, 8, and 12 was significantly higher when one or more parents had graduated from college and when home environments were more advantaged, regardless of which school those children attended. While science achievement of children from families of average SES or above remained consistently higher, the gap in achievement for poor children with or without these resources widened between 4th and 12th grades. Programs and policies aimed at improving achievement of students at risk because of poverty are unlikely to close the gap between more and less advantaged students. Without access to interventions and programs that compensate for a scarcity of academic resources at home, however, that gap may widen.

Parallel studies on children in highly stressed urban environments reported that expectations and beliefs operate as protective factors that moderated the resilience or vulnerability of affected individuals (Clausen, 1991; Cowen et al., 1992; Israelashvili, 1997; Wyman et al., 1993). Results of this study confirm that science achievement is higher for students who like science and believe they can do well regardless of their demographic status or academic acumen. The protective effects of positive attitudes may be especially important for minority students and girls as they move from elementary school to high school and beyond. The science achievement of minority students and girls who did not perceive that science was a subject that they could enjoy and succeed at learning was significantly lower than that of their counterparts who felt otherwise. Although inferences about causal relationships between attitudes and achievement are unwarranted, the observation should spark discussion about the feelings of students at risk and generate ideas for enhancing the protective effects of this personal attribute so that the benefits are more similar to those observed for students with lower probabilities of academic risk.

Academic resilience of minority students may be particularly sensitive to the opportunities structured by schools through policies that influence teachers' instructional choices. Access to high-quality science instruction is unlikely to close the gap in achievement, in part, because majority students in the same schools appear to benefit more from those opportunities. However, the findings support policies aimed at changing the way that science has been taught traditionally in high school. By 12th grade, the risks associated with minority status were significantly lower when minority students had access to a rich array of laboratory materials in their science classes.

Whereas these results should encourage thinking about how processes and policies that operate at the individual, home, and school levels of a child's educational environment moderate science achievement, causal inferences based on results of the analyses are unjustified. Policy makers and educators can use these findings to stimulate discussion about the significance of social context and sources of variation for students at risk and to suggest ways to use NAEP science proficiency data to further understanding of risk and resilience.

NOTE

1. Academic resilience is the demonstration of academic achievement that is higher than what would he predicted for someone at risk for low achievement because of demographic status.

REFERENCES

Anderson, O. R. (1998). A neurocognitive perspective un current learning theory and science instructional strategies. Science Education, 81, 67 89.

Bender, D., & Losel, F. (1997). Protective risk effects of peer relations and social support on antisocial behavior in adolescents from multi-problem milieus. Journal of Adolescence, 20, 661-678.

Blocker, L. S., & Copeland, E. P. (1994). Determinants of resilience in high-stressed youth. The High School Journal, 77, 286- 295.

Bock, R. D. (Ed.). (1989). Multilevel analysis of educational data. San Diego: Academic Press.

Brooks, R. B. (1994). Children at risk: Fostering resilience and hope. American Journal of Onhopsychiatry, 64, 545-55 3.

Bryk, A., Raudenhush, S., & Congdon, R. (1996). Hierarchical linear and nonlinear modeling with HLM/2L and / HLM/3L /migrants. Chicago: Scientific Software International.

Bryk, A. S., & Raudenhush, S. W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101, 147-158.

Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

Burkam, D. T., Lee, V. E., & Smerdon, B. A. (1997). Gender and science learning early in high school: Subject matter and laboratory experiences. American Educational Research Journal, 34(2), 297-331.

Burstein, L. (1980). The analysis of multilevel data in educational research and evaluation. In D. Berliner (Ed.), Review of research in education (pp. 158-255). Washington, DC: American Educational Research Association.

Burstein, L., Linn, R. L., & Capell, F.J. (1978). Analyzing multilevel data in the presence of heterogeneous within-class regressions. Journal of Educational Statistics, 3, 547-385.

Bush, D. M., & Simmons, R. G. (1987). Gender and coping with the entry into early adolescence. In R. G. Barnett, L. Biener, & Ci. Baruch (Eds.), Gender and stress. New York: Free Press.

Carey, N., & Shavelson, R. ( 1988). Outcomes, achievement, participation, and attitudes. In R. J. Shavelson, L. M. McDonnell, & J. Oakes (Eds.), Indicators for monitoring mathematics and science education (pp. 147-191). Los Angeles: Rand Corporation.

Carey, S. (1985). Conceptual change in childhood. (Cambridge, MA: MIT Press.

Carmichael, P., Driver, R., Holding, 15., Phillips, I., Twigger, D., & Watts, M. (1990). Research on students' conceptions in science: A bibliography. University of Leeds, UK: Centre for Studies in Science and Mathematics Education.

Catterall, J. S. (1998). Risk and resilience in student transitions to high school. American Journal of Education, 106, 302- 555.

Clausen, J. S. ( 1991 ). Adolescent competence and the shaping of the life course. American Journal of Sociology, 96, 805-842.

Clewell, B. C., Anderson, B. T., & Thorpe, M. E. (1992). Breaking the harriers: Helping female and minority students succeed in mathematics and science. San Francisco: Jossey-Bass.

Cohen, J. (1998). Statistical power analysis for the behavioral sciences (2nd ed., pp. 109-143). Hillsdale, NJ: Erlbaum.

Coleman, J. S., Camphell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, E D., et al. (1966). Equality of educational opportunity. Washington, DC: U.S. Government Printing Office.

Coleman, J. S., & Hoffer, T (1987). Public and private high schools: The impact of communities. New York: Basic Books.

Compas, B. E., Howcll, D. C., Phares, V, Williams, R. A., & Giunta, C. T. (1989). Risk factors for emotional/behavioral problems in young adolescents: A prospective analysis of adolescent and parental stress symptoms. Journal of Consulting and Clinical Psychology, 57, 732-740.

Cowen, E. L., Work, W. C., & Wyman, P.A. (1992). Resilience among profoundly stressed urban children. In M. Kessler & S. E. Goldston (Eds.), The present and future of prevention: In honor of George Albee (pp. 155-168). San Francisco: Sage.

Czikszentmihallyi, M., & Larson, R. (1984). Being adolescent. New York: Basic Books.

Donmoyer, R. (1995). Rhetoric and reality of systemic reform: A critique of the proposed national science education standards. Cognosos, 4(1), 1-3, 7-8.

Ertepinar, H., & Geban, O. (1996). Effect of instruction supplied with the investigative-oriented laboratory approach on achievement in a science course. Educational Researcher, 38(3), 333-341.

Floyd, C. ( 1996). Achievingdespite the odds: A study of resilience among a group of African American high school seniors, journal of Negro Education, 65, 181-189.

Freedman, M. P. (1997). Relationship among laboratory instruction, attitude toward science, and achievement in science knowledge. Journal of Research in Science Teaching, 34(40), 343- 357.

Freiberg, J. H. (1993). A school that fosters resilience in inner- city youth. Journal of Negro Education, 62, 364-376.

Garmezy, N. (1983). Stressors of childhood. In N. Garmezy & M. Ruttcr, (Eds.), Stress, coping, and development in children (pp. 43- 84). New York: McGraw-Hill.

Garmezy, N. (1985). Stress-resistant children: The search for protective factors. In J. E. Stevenson (Ed.), Recent research in developmental psychopathology. Journal of Child Psychology and Psychiatry (Book Supplement No. 4, pp. 213-233). Oxford, England: Pergamon Press.

Garmezy, N. (1991). Resiliency and vulnerability to adverse developmental outcomes associated with poverty. American Behavioral Scientist, 34, 417-430.

Garmezy, N., & Masten, A. S. (1986). Stress, competence, and resilience: Common frontiers for therapist and psychopathologist. Behavior Therapy, 17, 500-521.

Garmezy, N., Masten, A. S., & Tellegen, A. (1984). The study of stress and competence in children: A building block for developmental psychopathology. Child Development, 55, 97-111.

George, R., & Kaplan, D. (1998). A structural model of parent and teacher influences on science attitudes of eighth graders: Evidence from NELS:88. Science Education, 82, 93-109.

Gibbons, A. (1992). Growing scientists for the 21st century. Science, 258, 1195.

Glasson, G. E. (1989). The effects of hands-on and teacher demonstration laboratory methods on science achievement in relation to reasoning ability and prior knowledge. Journal of Research in Science Teaching, 26(2), 121-131.

Goldstein, H. (1987). Multilevel models in educational and social research. New York: Oxford University Press.

Gonzalez, R., & Padilla, A. M. (1997). The academic resilience of Mexican-American high school students. Hispanic Journal of Behavioral Sciences, 19, 301-317.

Gore, S., & Eckenrode, J. (1994). Context and process in research on risk and resilience. In R. J. Haggerty, L. R. Sherrod, N. Garmezy, & M. Rutter (Eds.), Stress, risk, and resilience in children and adolescents: Processes, mechanisms, and interventions. New York: Cambridge University Press.

Greenfield, T. A. (1996). Gender, ethnicity, science achievement, and attitudes. Journal of Research in Science Teaching, 33, 901- 933.

Grossman, R. K., Beinashowitz, J., Anderson, L., Sarurai, M., Finnin, L., & Flaherty, M. (1992). Risk and resilience in young adolescents. Journal of Youth and Adolescence, 21, 529-550.

Hilton, T. L., & Lee, V. E. (1988). Student interest and persistence in science. Journal of Higher Education, 59(5), 510- 526.

Hoffer, T. B., Rasinski, K. A., & Moore, W (1995). Social background differences in high school mathematics and science coursetaking and achievement. Washington, DC: U.S. Department of Education.

Israelashvili, M. (1997). School adjustment, school membership and adolescents' future expectations. Journal of Adolescence, 20, 525-535.

Jackson, S., Born, M., & Jacob, M. (1997). Reflections on risk and resilience in adolescence. Journal of Adolescence, 20, 609-616.

Johnson, E. G., Qian, J., Wallace, L., & Rust, K. F. (1999.) Weighting procedures and estimation of sampling variance. In N. L. Allen, J. E. Carlson, & C. A, Zclcmik (Eds.), The NAEP 1996 technical report (NCES 199-452, pp. 197-234). Washington, OC: U.S. Department of Education.

Kreft, I., & De Leeuw, J. (1998). Introducing multilevel modeling. London: Sage.

Larson, R., & Asmussen, L. (1991). Anger, worry, and hurt in early adolescence: An enlarging world of negative emotions. In M. E. Colten & S. Gore (Eds.), Adolescent stress: Causes and consequences. New York: Aldine de Gruyter.

Lee, V. E., Chen, X., & Smerdon, B. A. (1996). The influence of school climate on gender differences in the achievement and engagement of young adolescents. Washington, DC: American Association of University Women.

Luthar, S. S. (1991). Vulnerability and resilience: A study of high-risk adolescents. Child Development, 62, 600-616.

Luthar, S. S., & Zigler, E. (1991). Vulnerability and competence: A review of research on resilience in childhood. American Journal of Orthopsychiatry, 61, 6-22.

Lynn, R. (1977). The intelligence of the Japanese. Bulletin of the British Psychological Society, 40, 464-468.

Madigan, T. (1997). Science proficiency and course taking in high school: The relationship of course-taking patterns to increases in science proficiency between 8th and 12th grades. Washington, DC: U.S. Department of Education.

Mager, R. (1968). Developing attitude toward learning. Belmont, CA: Fearon.

Mason, C. L., & Kahle, J. B. (1989). Student attitudes toward science and science-related careers: A program designed to promote a stimulating gender-free learning environment. Journal of Research in Science Teaching, 26(1), 25-39.

Masten, A. S., & Garmezy, N. (1985). Risk, vulnerability, and protective factors in developmental psychopathology. In B. B. Lahey & A. E. Kazdin (Eds.), Advances in clinical child psychology (Vol. 8, pp. 1-52). New York: Plenum Press.

Masten, A. S., Garmezy, N., Tellegen, A., Pellcgrini, D. S., Larkin, K., & Larsen, A. ( 1988). Competence and stress in school children: The moderating effects of individual and family qualities. Journal of Child Psychology and Psychiatry, 29, 745-764.

McCauley, K. J. (1995). A restructured biology classroom. Teaching and Change, 2(2), 152-168.

Mickelson, R. (1990). The attitude-achievement paradox among black adolescents. Sociology of Education, 63, 44-61.

Mislevy, R. J., Johnson, E. G., & Muraki, E. (1992). Scaling procedures in NAEP. Journal of Educational Statistics, 17(92), 131- 154.

Mordkowitz, E., & Ginsburg, H. (1987). Early academic socialization of successful Asian-American college students. Quarterly Newsletter of the Laboratory of Comparative Human Cognition, 9, 85-91.

National Center for Education Statistics. (1992). National educational study of 1988: A profile of American eighth-grade mathematics and science instruction (Statistical Analysis Report, NCECS 92-486). Washington, DC: U.S. Department of Education.

National Research Council. (1996). National science education standards. Washington, DC: National Academy Press.

Nettles, S. M., & Pleck, J. H. (1994). Black adolescents in the United States. In R. J. Haggerty, L. R. Sherrod, N. Garmezy, & M. Rutter (Eds.), Stress, risk, and resilience in children and adolescents: Processes, mechanisms, and interventions (pp. 147- 181). New York: Cambridge University Press.

Norwich, B., & Duncan, J. (1990). Attitudes, subjective norm, perceived preventive factors, intentions and learning science: Testing a modified theory of reasoned action. British Journal of Educational Psychology, 60, 312-321.

Odubunmi, O., & Belogun, T. A. (1991). The effect of laboratory and lecture teaching methods on cognitive achievement in integrated science. Journal of Research in Science Teaching, 28(3), 213-234.

Ogbu, J. (1978). Minority education and caste. San Diego: Academic Press.

Ogbu, J. (1992). Understanding cultural diversity and learning. Educational Researcher, 12, 5-14.

Ogbu, J. U. (1991). Minority coping responses and school experience. Journal of Psychohistory, 18, 433-456.

O'Sullivan, C. Y., & Weiss, A. R. ( 1999). Student work and teacher practices in science: A report on what students know and can do (NCES 1999-455). Washington, DC: U.S. Department of Education,

Peng, S. S., & Wright, D. (1994). Explanation of academic achievement of Asian-American students. The Journal of Educational Research, 87, 346-352.

Piaget, J. (1970). Genetic epistemology. New York: Columbia University Press.

Radke-Yarrow, M., & Sherman, T. (1990). Hard-growing: Children who survive. In J. Rolf, A. S. Masten, D. Cicchetti, K. Neuchterlein, & S. Weintrauh (Eds.), Risk and protective factors in the development of psychopathology. New York: Cambridge University Press.

Raizen, S. A., & Jones, L. V. (1985). Indicators of precollege education in science and mathematics: A preliminary review. Washington, DC: National Academy Press.

Rak, C. E, & Patterson, L. E. (1996). Promoting resilience in at- risk children. Journal of Counseling and Development, 74, 368-373.

Reynolds, A. J. (1998). Resilience among black urban youth: Prevalence, intervention effects, and mechanisms of influence. American Journal of Onhopsychiatry, 68, 84-100.

Richters, J., & Weintrauh, S. (1990). Beyond diathesis: Toward an understanding of high-risk environments. In J. Rolf, A. S. Masten, D. Cicchetti, K. Neuchiertein, & S. Weintrauh (Eds.), Risk and protective factors in the development of psychopathology. Cambridge, England: Cambridge University Press.

Ricchard, D. E. (1994). National science education standards: Around the reform bush . . . again? Clearing House, 67, 135-136.

Rodriguez, A. J. (1997). The dangerous discourse of invisibility: A critique of the National Research Council's national science education standards. Journal of Research in Science Teaching, 34(1), 19-37.

Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: Wiley.

Rutter, M. (1979). Protective factors in children's responses to stress and disadvantage. In M. W. Kent & J. E. Rolf (Eds.), Primary prevention of psychopathology, Vol. 3: Social competence in children. Hanover, NH: University Press of New England.

Rutter, M. (1985). Resilience in the lace of adversity: Protective factors and resistance to psychiatric disorders. British Journal of Psychiatry, 147, 598-611.

Rutter, M. (1987). Psychosocial resilience and protective mechanisms. American Journal of Orthopsychiatry, 57, 316-331.

Rutter, M. (1994). Stress research: Accomplishments and tasks ahead. In R. J. Haggerry, E. R. Sherrod, N. Carmez\y, & M. Rutter (Eds.), Stress, risk, and resilience in children and adolescents: Processes, mechanisms, and interventions (pp. 354-386). New York: Cambridge University Press.

Seifer, R., & Sameroff, A. J. (1987). Multiple determinants of risk and invulnerability. In E. J. Anthony & B. J. Cohler (Eds.), The invulnerable child (pp. 51-69). New York: Guilford Press.

Skolnick, J., Langbort, C., & Day, L. (1982). How to encourage girls in math and science. Englewood Cliffs, NJ: Prentice-Hall.

Stohr-Hunt, P. M. (1996). An analysis of frequency of bands-on experience and science achievement. Tournai of Research in Science Teaching, 33(1), 101-109.

Sue, S., & Okazaki, S. (1990). Asian-American educational achievement: A phenomenon in search oi an explanation. American Psychologist, 45, 913-920.

van Welzenis, 1. (1997). The self-concept of societally vulnerable and delinquent boys within the context of school and leisure activities. Journal of Adolescence, 20, 695-705.

von Glaserfeld, E. (1984). An introduction to racial constructivism. In P. Watzlawick (Ed.), The invented reality (pp. 17- 40). New York: Norton.

von Glaserfetd, E. (1987). Learning as a constructive activity. In C. ]anvier (Ed.), Problems of representation in the leaching and learning of mathematics (pp. 3-17). Hillsdalc, NJ: Erlbaum.

Von seeker, C. (2002). Effects of inquiry-based teacher practices on science excellence and equity. The journal oj Educational Research, 9.5, 151-160.

Von seeker, C, & Lissiez, R. W. (1999). Estimating the impact of instructional practices on student achievement in science. Journal oj Research m Science Teaching, 36( 10), 1110-1126.

Wang, M. C., & Gordon, E. W. (1994). Educational resilience in inner-city America. Mahwah, NJ: Erlbaum.

Werner, E. E. (1989). High risk children in young adulthood: A longitudinal study from birth to 32 years. American Journal of Ortholpsychialry, .59,72-81.

Werner, E. E., & Smith, R. S. (1982). Vulnerable but invincible: A longitudinal study of resilient children and youth. New York: McGraw-Hill.

Werner, E. E., & Smith, R. S. (1992). Overcoming the odds: High risk children from birth to adulthood. Ithaca, NY: Cornell University Press.

Wyman, P A., Cowen, E. L., Work, W. C., & Kerley, J. H. (1993). The role ot children's future expectations in self-esteem functioning and adjustment to life-stress: A prospective study of urban at-risk children. Development and PsycholMlhology, .5, 649- 661.

CLARE VON SECKER

University of Maryland

Address correspondence to Clare Von Secker, Department of Measurement, Statistics and Evaluation, College of Education, University of Maryland, College Park, MD 20742. (E-mail: cv42@umail.umd.edu)

Copyright HELDREF PUBLICATIONS Nov/Dec 2004

Source: Journal of Educational Research, The

More News in this Category