• E-mail
• Print
• Comment
• Font Size
• Digg
• del.icio.us
• Discuss article

Slowing the Influence of Flawed Mathematics and Science Education Studies

Posted on: Saturday, 15 January 2005, 03:00 CST

In school science and mathematics, we sometimes judge the value of a research study by the prestige of the publication printing the study and how often we heard others bolster their arguments by citing such research. These strategies ostensibly help us determine which studies have merit and valid fodder to construct follow-up research studies and practitioner teaching approaches.

But the historical record argues against depending on such gatekeeping strategies. The problem is that the history of mathematics and science education is repeating itself, for we (mathematics and science educators) habitually accept the validity of talked-about studies instead of taking time to scrutinize them - those studies with the greatest chance of influencing teaching approaches. This somewhat reckless disposition on our part, in turn, may muddle future research agendas, national funding decision- making, and teachers' classroom practices. I describe major troubles with three highly visible, influential studies, each painting bleak pictures, each from a different decade and each with increasingly difficult-to-identify problems.

Each study was apparently authored by bright, honest scholars who are sincerely concerned about improving science and mathematics education. But they each produced "evidence-based" inferences derived from opinions and questionable data or fatally flawed research methods. If a research method is flawed, logically, the data and resulting inferences are flawed. Whether each study's inferences are valid (i.e., true) remains a matter of conjecture. Readers must remain vigilant by continually asking such questions as, Where is the evidence and how was it derived?

A Nation At Risk

Most of us mathematics and science educators either read or believed forceful commentators' reactions to the 1983 influential publication, A Nation at Risk (National Commission on Excellence in Education, 1983), an inspiring change-agent document. To our chagrin, many of us later discovered after reading Berliner and Biddle's (1995) American Educational Research Association's (AERA) award-winning book, The Manufactured Crisis, that the Nation report was full of opinions and "simplistic misleading generalizations" (Berliner & Biddle, 1995, p.3) The authors characterized the so- called "failures" of school mathematics and science programs without providing readers with reasonable arguments linked to evidence. Since no data were collected by the authors of this downbeat study (Berliner & Biddle, 1995), there are no methods to assess. Public education in the U.S. may have numerous problems in mathematics and science education, but this document failed to argue the case successfully.

Problems With TIMSS

Last decade a respectable, albeit underfunded, group of researchers published the Third International Mathematics and Science Study (TIMSS; U.S. Department of Education, 1998) comparing science and mathematics achievement of students living in different countries, concluding - according to some other commentators - that U.S. students were receiving mediocre education in mathematics and science compared to students residing in other selected countries. The study and later some of the authors' public comments rightly described a few of the limitations inherent in their own research methodologies. The problem with TIMSS was not the work as much as the highly visible commentators drawing erroneous conclusions and failing to highlight the study's severe methodological problems. (For additional details, see Holliday &Holliday, 2003)

First, many of the TIMSS participating nations had limited financial resources and perhaps lacked the political courage needed to produce representative samplings of their nation's student population. In addition, only 5 (Czech Republic, Hungary, New Zealand, Sweden, and Switzerland) of the 21 participating nations even claimed to sample their students using the standards set by the TIMSS researchers. The United States failed to meet the TIMSS's sampling standards. So one wonders how the data presented in TIMSS can be taken seriously by educators still citing the TIMSS.

Second, making curricular comparisons about mathematics (and science) curriculums across a large country like the United States is nearly impossible for methodological reasons (Kilpatrick, 2003), but comparing student performances on tests of questionable equivalency written in different languages and embedded in differing political and educational systems focusing on varying educational goals is a challenge beyond one's imagination. For example, did the TIMSS researchers use a standard back-translation procedure to support the precision of the translation of test items into many languages (Wang & Guthrie, 2004), one of the first steps in establishing culture equivalency?

Still, TIMSS keeps showing up in numerous recent research articles in mathematics and science teaching journals authored by excellent researchers and published in excellent refereed outlets. During the past few years, some science education articles published in the Journal of Research in Science Teaching (Clark & Jorde, 2003; Siegal & Ranney, 2003; van Uriel, Beijaard, & Verloop, 2001) and four other articles published in Science Education (Davis, 2002; Ogawa, 2001; Zady, Portes, & Oches, 2002; Zubrowski, 2002) used TIMSS data or other people's interpretations of TIMSS data to support their otherwise well-reasoned arguments to explore important research questions in science education. Editors of these journals, unfortunately, are at the mercy of external reviewers of varying competencies and publishing standards.

Johnny Lott, a past-president of the National Council of Teachers of Mathematics (NCTM) and a university professor of mathematics, supported one interpretation drawn from one of the TIMSS surveys assessing students residing in 38 countries. From this TIMSS data, he encourage educators to follow Japan's curriculum model by using an "integrated" approach, a model that also inspired some leading Georgia educators to change the state's approach to mathematic education (Galley, 2004).

The notion of integrating related school subjects in mathematics and science may be valid and keeps popping up decade after decade for a mixture of good and flawed reasons. The problem here is that people cited an international comparative study like TIMSS to support their rationale. Perhaps mathematics and science teaching programs are truly inadequate compared to other selected countries, and increasingly integrating mathematics and science is a good idea, but using the TIMSS to bolster such notions remains unpersuasive.

Troubles With AAAS's TextbookEvaluation Study

The American Association for the Advancement of Science (AAAS) is the largest and most politically powerful organization of scientists and engineers in the world and a group deeply and sincerely concerned with the quality of mathematics and science teaching in our schools. Nevertheless, serious problems with AAAS's research methods published in an award-winning article were unveiled, suggesting that they unknowingly designed a study over the years bound to make practitioners' textbooks look unacceptable, according to a follow-up commentary (Holliday, 2003a) and a newly released 2003 AAAS document described in their rejoinder (Kesidou, & Roseman, 2003, p. 536).

Kesidou and Roseman (2002) in their work at the AAAS reported ratings of middle school science textbooks by rating nine popular programs. (AAAS has also evaluated mathematics textbook-based programs in separate studies.) Results: These science textbooks are unsatisfactory. (For additional details and references, see Holliday 2003a, b) This award-winning AAAS study (recipient of the 2003 Journal of Research in Science Teaching award bestowed by the National Association for Research in Science Teaching) in one sense is part of a long-running campaign by AAAS disparaging the quality of publishers' mathematics and science textbooks.

But methodological problems with this study are enormous, yet not evidently obvious to most scholars. First, raters evaluated the science textbooks after receiving seven days of still undefined training by AAAS staff. (Holliday, 2003a, b) Second, the AAAS- developed rating criteria used to judge textbook quality was derived from nearly equivalent criteria located in a 1996 AAAS paper on their website, coupled later to unrealistic, highly restrictive, qualifying language (Holliday, 2003a, b). Third, the textbook raters' guidelines made it virtually improbable for any popular textbook usable by ordinary practicing science teachers to pass the AAAS science textbook test (Holliday, 2003 a, b)

Fourth, the AAAS's authors made unwarranted generalizations in both research articles, even if the method had been flawless. For example, Kesidou and Roseman (2003, p. 542) generalized their highly limited, focused findings by saying, "Although we, too, would have welcomed more positive findings [regarding the assessed quality of the science textbooks] from our study, the findings themselves offer a clear picture of what needs to be done to improve materials in the future" (see Holliday, 2003b). Further explanations were not provided by the authors.

Fifth, it is not clear how Kesidou \and Roseman (2002) made the leap from scaled labels of "excellent," satisfactory," or "poor" to alternative scaled values, "0-1 = poor; 1.5= fair; 2 = satisfactory; 2.5 = very good; 3 = excellent" (Kesidou, & Roseman, 2002, pp. 528- 529). The raters' guideline rating schema (see Kesidou, & Roseman, 2003, p. 536 for website address or Holliday, 2003b) format differed among the 22 guidelines within the same document. The authors did not even correctly copy most of the guidelines from their AAAS original documents presented on their website to their writing of the award-winning article's appendix.

Moreover, guideline number four presented on the website included an extra level of textbook competency called "fair," along with the other three standard ratings of excellent, satisfactory or unsatisfactory, for no stated reasons. In addition, the rating scheme for guidelines numbers 20, 21, and 22 lacked any rating schema, with no levels of evaluation for no apparent reason. Individual rating schema, moreover, were not rationalized or linked to research (Holliday, 2003b).

Sixth, it was not stated how the series of mini book reports characterizing the textbooks and occupying most of the results section (7 journal pages out of 11) of the Kesidou and Roseman (2002, p. 527-538) research were compiled and derived. This is unusual because readers expect an article's results section to present the findings of the research study detailed in its method section, not a major presentation of results derived from some other study not described in the article's method section (Holliday, 2003a).

AAAS has just published a similar study (Stern & Roseman, 2004), in which they continue to make most of the same scientific errors, failing to describe their research methods adequately, apparently using the flawed methods reported in their 2002 award-winning study, and still engaging in careless scientific thinking. (Compare Holliday, 2003a, b, with Kesidou & Roseman, 2002,2003). Also, notice how the AAAS authors (Stern & Roseman, 2004) mainly referred readers back to their earlier problematic documents as assessed by Holliday (2003a, b). Yet, an unanswered empirical question remains: Is there evidence of students not learning important science and mathematics from these textbooks? Today, we seem to have too many untested hypotheses and lots of shaky data-based opinions.

Very few educators talk about the fact that around 90% of surveyed mathematics and science teachers rank their textbooks as "fair,""good,""very good," or "excellent," in a widely publicized study by Weiss (2001), who used an exemplary research method. Other surveyed teacher ranked their textbooks as "poor" or "very poor." (Weiss, 2001, p. 87, Table 6.8) Of course, such rankings by surveyed teachers fail to validate their value of these textbook-based programs (which are far from perfect), as correctly suggested by Weiss. Unfortunately, the AAAS study failed to shed much light on existing hypotheses about mathematics and science textbooks, potentially helpful to publishers, researchers and teachers.

Conclusion

Most researchers (including this author) have surely cited some studies based in weak methodologies to buttress rationales supporting their research works or grant proposals. The point is that all of us in mathematics and science teaching should work hard to avoid using such poorly conceived, yet popular, studies because researchers are more likely to diminish the power of their rationales when their papers are read by careful readers.

Careful readers evaluate the validity and specific evidence for each methodological and other concerns potentially embedded in influential studies like those under review by examining the study, follow-up commentaries and accessible supporting documents. Resolution of these concerns by readers may impact future curricular research and tens of millions of dollars in future funding allocations. Mathematics and science educators, including classroom teachers, must remain vigilant and willing to challenge the validity of research rather than depending on unreliable gatekeeping procedures. The question remains: Will federal funding organizations, researchers' agendas, educational policymakers, and teachers continue to be influenced by such flawed research studies?

References

Berliner, D. C., & B. J. Biddle. 1995. The manufactured crisis. Reading, MA: Addison-Wesley.

Clark, D., & Jorde, D. (2003). Helping students revise disruptive experimentally supported ideas about thermodynamics: Computer visualizations and tactile models. Journal of Research in Science Teaching, 41, 1-23.

Davis, K. S. (2002). "Change is hard": What science teachers are telling us about reform and teacher learning of innovation practices. Science Education, 87, 3-30.

Galley, M. (2004). Georgia reaches out to Japan for math- curriculum model. Education Week, 23(27), 13.

Holliday, W. G. (2003a). Methodological concerns about AAAS's Project 2061 study of science textbooks. Journal of Research in Science Teaching, 40, 529-534. See the highly similar document: http://www.education.umd.edu/news/ hollidayaaastextbooks.pdf

Holliday, W. G. (2003b). New methodological concerns about AAAS's Project 2061 study of science textbooks. Unpublished manuscript, University of Maryland [Online]. Available: http:// www.education.umd.edu/news/ hollidayaaasfollow-up.pdf

Holliday, W. G., & Holliday, B. W. (2003). Why using international comparative mathematics and science achievement data from TIMSS is not helpful. The Education Forum, 67, 250-257. See the publisher's document: http:// www.education.umd.edu/news/ timssarticleholliday.pdf

Kesidou, S., & Roseman, J. E. (2002). How well do middle school science programs measure up? Findings from Project 2061 's curriculum review. Journal of Research in Science Teaching, 39, 522- 549.

Kesidou, S., & Roseman, J. E. (2003). Project 2061 analyses of middle-school science textbooks: A response to Holliday. Journal of Research in Science Teaching, 40, 535-543.

Kilpatrick, J. (2003). What works? In S. L. Senk & D. R. Thompson (Eds.), Standards-based school mathematics curricula: What are they? What do students learn? (pp. 471-488). Mahwah, NJ: Lawrence Erlbaum Associates.

National Commission on Excellence in Education. ( 1983). A nation at risk: The imperatives for educational reform. Washington, DC: U.S. Department of Education.

Siegel, M. A., & Ranney, M. A. (2003). Developing the changes in attitude about the relevance of science (CARS) questionnaire and assessing two high school science classes. Journal of Research in Science Teaching.. 40, 757-775.

Stern, L., & Roseman, J. E. (2004). Can middle-school science textbooks help students learn important ideas? Findings from Project 2061's curriculum evaluation study: Life Science. Journal of Research in Science Teaching., 41, 538-568.

U.S. Department of Education. (1998). Pursuing excellence: A study of U.S. twelfth-grade mathematics and science achievement in international context, initial findings from the Third International Mathematics and Science Study, Washington, DC: National Center for Education Statistics, Office of Educational Research and Improvement.

vanDriel, J. H., Beijaard, D., & Verloop, N. (2001). Professional development and reform in science education: The role of teachers' practical knowledge. Journalof Research in Science Teaching, 38, 137- 158.

Wang, J. H-Y, & Guthrie, J. T. (2004). Modeling the effects of intrinsic motivating, extrinsic motivation, amount of reading, and past reading achievement on text comprehension between U.S. and Chinese students. Reading Research Quarterly, 39, 162-186.

Weiss I. R. (2001) Report of the 2000 National Survey of Science and Mathematics Education [Online]. Available: http:// 2000survey.horizon-research.com/reports/status/complete.pdf

Zady, M. F., Portes, P. R., & Oches, V. D. (2003). Examining classroom interactions related to difference in students' science achievement. Science Education., 87, 40-63.

Zubrowski, B. (2002). A curriculum framework based on archetypal phenomena and technologies. Science Education, 86, 481-501.

William G. Holliday

GUEST EDITORIAL

Copyright School Science and Mathematics Association, Incorporated Jan 2005

Source: School Science and Mathematics

More News in this Category