###### February 1, 2004

# Relationship Between Professional Development, Teachers’ Instructional Practices, and the Achievement of Students in Science and Mathematics

The purpose of this study was to examine the relationship between different types of professional development, teachers' instructional practices, and the achievement of students in science and mathematics. The types of professional development studied included immersion, examining practice, curriculum implementation, curriculum development, and collaborative work. Data regarding teachers' instructional practices and the amount of professional development were collected using teacher surveys. Ninety-four middle school science teachers and 104 middle school mathematics teachers participated in the study. Student achievement was measured using eighth grade state science and mathematics achievement test data. Regression analyses suggested that for both science and mathematics teachers, examining practice and curriculum development were significantly related to the use of standards-based instructional practices. Only curriculum development for mathematics teachers was significantly related to student achievement. Implications of results for the professional development of science and mathematics teachers are discussed.

Over the years there have been numerous calls for educational reform that uses professional development as a central feature of improving science and mathematics education (Elmore, 1996; National Commission on Mathematics and Science Teaching, 2000; National Education Goals Panel, 1994). The need for professional development is great, given the large number of science and mathematics teachers who enter the teaching profession on emergency certification and the number of science and mathematics teachers who teach out of field (National Commission on Teaching and America's Future, 1996). Despite the need, there are some who question the investment in professional development for teachers unless it clearly results in improved student achievement (Education Commission of theStates, 1997;Riley, 199 8). The accountability movement in the United States has also placed increased pressure on schools and districts to provide targeted professional development that will clearly help improve student achievement.

Unfortunately, research regarding the impact of professional development on student achievement is limited. The majority of research on professional development has, instead, examined changes in instructional practices, teachers' knowledge, teachers' beliefs, and other important variables that may be indirectly linked to student achievement (Loucks-Horsley & Matsumoto, 1999). These are important variables to examine even if they are not directly linked to student achievement. However, more research is needed that examines the relationship between professional development and student achievement.Research on the impact of professional development on student achievement is limited because it is difficult and expensive to study, and the link between professional development and student achievement is complex. Models have been proposed to illustrate the complexity and number of possible factors that can impinge upon the relationship between professional development and student achievement. In a review of research on professional development in science and mathematics, Loucks-Horsley and Matsumoto (1999) proposed a model of professional development in which high quality professional development was proposed to lead to improved teacher learning (i.e., knowledge, skills, and beliefs), which in turn, would lead to improved student achievement. Loucks-Horsley, Hewson, Love, and Stiles (1998) also identified numerous professional development strategies for science and mathematics teachers, which they describe as high quality professional development. The strategies can be roughly grouped into five categories: immersion, examining practice, curriculum development, curriculum implementation, and collaborative work.

* Immersion strategies involve having teachers actually "do" science or mathematics and gain the experience of doing science or math with a scientist or mathematician.

* Curriculum implementation involves having teachers using and refining the use of instructional materials in the classroom.

* Curriculum development involves having teachers help create new instructional materials to better meet the needs of students.

* Examining practice includes case discussion of classroom scenarios or examining real classroom instruction.

* Collaborative work includes study groups, peer coaching, mentoring and classroom observation and feedback.

Theoretically, these five types of professional development fit with Ball and Cohen's ( 1999) "practicebased" theory of professional development. According to this theory, professional learning for teachers should emphasize long-term active engagement, connections between teachers' work and their own students' learning, and opportunities to practice and apply what students learns in a real- world context. The emphasis is on a continuous cycle of exploring new issues and problems, creating cognitive dissonance, engaging in collaborative discussions, constructingnew understanding, and improving professional practice.

The types of professional development described by Loucks- Horsley et al. (1998) have not been extensively researched. Furthermore, the link between these different approaches and student achievement is not well established. Several of these professional development approaches, however, have been shown to affect positively teachers' knowledge and instructional practice. For example, research on examining practice suggests it can help teachers improve their knowledge of mathematics and help change their instructional practice (Barnett, 1998; Schifter, 1996). Curriculum implementation has also been found to improve teachers' knowledge of mathematics and instruct'on (Cohen & Hill, 1998). These results provide some evidence for the efficacy of these approaches in the professional development of science and mathematics teachers. It would be helpful to the field to compare and contrast these different approaches to understand better the relative merits of these professional development strategies. This information could help providers ofprofessional development choose between different strategies and plan more effective professional development for science and mathematics teachers.

Purpose of the Study

The purpose of this study was to examine the relationship between different types of professional development, teachers' instructional practices, and the achievement of their students in science and mathematics. Specifically, this study examined five types of professional development recommended by LoucksHorsleyetal. (1998) and their relationship to teachers' instructional practices and the achievement of students.

Research Design

This study was part of a larger study funded by the National Science Foundation (NSF) to examine the impact of state reform efforts in science and mathematics on student achievement. The authors were funded to conduct an independent evaluation of the reform and were not part of the design or implementation of any initiative.

The state in which this study took place was attempting to reform science and mathematics education through the use of coordinated professional development workshops. The goal was to reform science and mathematics education by encouraging more science and mathematics teachers to engage in long-term intensive professional development. The rationale was that teachers have the most direct effect on classroom instruction and, hence, have the greatest opportunity to affect student achievement.

A wide variety ofprofessional development workshops were offered throughout the state in both mathematics and science. University faculty offered workshops, and teachers could choose among different offerings with varied length and duration. They ranged from short 3- to 5-day workshops to longer summer workshops, with extended follow- up throughout the school year. For example, in one workshop for biology, chemistry, and physical science teachers the focus was on collecting and analyzing water collected from a local swamp. The teachers attended an initial 10-day summer workshop, followed by 5 additional days during the academic school year.

Given the voluntary nature of teachers' participation, the engagement of teachers varied quite widely, which meant that some teachers engaged in more professional development than others. This situation created a natural ex-post facto design with variability in the type and frequency of professional development in which teachers engaged.

Sample

For this study, the focus was on eighth-grade science and mathematics teachers in a southern state. Data were collected from 94 science teachers and 104 mathematics teachers located in a total of 46 schools. The schools were located throughout the state and in rural, suburban, and urban districts. Teachers in this state are some of the lowest paid in the country. In addition, the state suffers from the large number of teachers who teach out of field or do not have backgrounds in science or mathematics. Demographic characteristics of the teachers in this study are included in Table 1. The results sugges\t that, on the whole, the mathematics teachers in this sample had slightly more years of classroom teaching experience than did the science teachers.

This state also suffers from one of the highest poverty rates in the nation, with over 50% of all students receiving free or reduced- price lunches. In addition, eighth-grade student scores on the National Assessment of Educational Progress (NAEP) in both science and mathematics are among the lowest in the country. Demographic characteristics of the students are included in Table 2. The science and mathematics students who were from the same schools, of course, appeared quite similar in terms of gender, race, speaking English, and grades in school.

Table 1

Demographic Characteristics of Science and Mathematics Teachers

Table 2

Demographic Characteristics of Students

Instrumentation

The written survey used in this study measured teachers' views of the extent to which they used the instructional methods described in the National Science Education Standards (National Research Council, 1996) and the National Council of Teachers of Mathematics ( 1989) standards. To develop the survey, questions were selected from established sources, such as Horizon Research Incorporated ( 1997), for use in the NSF supported Local Systemic Change Proj ects, Third International Mathematics and Science Study (1996), andNewmann and Associates' (1997) surveys used to measure authentic pedagogy. The items were all Likert scaled and modified to focus specifically on the extent to which teachers used various standards-based techniques.

"Standards-based" was defined according to the national standards in mathematics and science. The national standards forteaching science and mathematics closely match this state's standards. To help ensure quality instrument development, initial drafts of the surveys were written, pilot tested, and revised by a team of university-level science and mathematics educators. In addition, a measurement specialist from the university was consulted on psychometric properties of the items. The pilot testing included one- on-one talk aloud protocols to help ensure that teachers properly interpreted questions on the survey. The data from the pilot tests were used to check individual items for appropriate mean scores and ranges. Factor analyses and reliability analyses were also conducted to confirm that teachers had consistent views of the items and to trim the surveys down to the items that were most reliable and that most closely matched the construct of standards-based instruction. Using Cronbach's alpha as a measure of reliability, the standards- based curriculum and instruction scale had acceptable reliability of .77 for mathematics teachers and .81 for science teachers. (See appendix for the items on the scale.)

The survey had teachers indicate how often they used various instructional methods and activities using the following rating scale: 1 = never or rarely, 2 = once a month, 3 = once a week, 4 = two or three times a week, 5 = daily. A mean score on the curriculum and instruction scale was calculated for each teacher. This scale was used as the dependent variable in the regression analyses described in the results section.

State Achievement Tests

Existing state achievement test scores were used to measure student achievement. Although achievement test scores can be distal measures of student achievement, these testscores are the primary measure used to judge school quality and the performance of schools in this state. For this reason alone it is important to study the relationship between professional development and these tests. Mathematics mean achievement scores were calculated for students in the mathematics teachers' classes, and science mean scores were calculated for students in science teachers' classes. These class mean scores were used as the dependent variable in the regression analyses.

The state achievement test scores used in this study were part of a criterion-referenced state assessment system designed to measure student achievement of the state standards. The tests are administered to all eighth-grade students in the spring of the academic year in the content areas of English, mathematics, science, and social studies. In addition, the tests are aligned with the state content standards in each subject area. The mathematics test is designed to measure student achievement in six areas: number and number relations, algebra, measurement, geometry, data analysis, and patterns, relations and functions. The test includes multiple- choice and open-ended formats in an effort to measure both traditional computation skills and more advanced problem solving skills. Concepts and skills are often assessed in realistic contexts and situations. The openended questions have problems with more than one possible solution or more than one path to a solution, and students are asked to explain their solution in writing. The science test is designed to measure student achievement in five areas: physical science, life science, earth and space science, science and the environment, and science as inquiry. Included are multiple- choice questions that assess concepts and skills in all five areas, short answer questions that assess the four content areas, and a comprehensive science task that assesses students' ability to engage in scientific inquiry.

Results

Teachers also completed survey questions to help describe the type and duration of the professional development in which they participated. The survey included five questions about the extent to which teachers engaged in different types of professional development. For each type of professional development, teachers were asked to indicate how long they engaged in the activities. The choices were not at all, less than 2 weeks, 2 to 4 weeks, 4 to 6 weeks, or continuing contact for 6 months or more. Table 3 includes abreakdown of the amount of time science and mathematics teachers engaged in different types of professional development. It is interesting to note that a slightly higher percentage of science teachers engaged in no professional development at all.

Regression analyses were conducted separately for both science and mathematics teachers. In addition, two different regression analyses were run on both science and mathematics teachers; one using teachers' scores on the curriculum and instruction scale as the dependent variable and one using student achievement as the dependent variable. The independent variables in all regressions were the five types of professional development (i.e., immersion, curriculum implementation, curriculum development, examining practice, and collaborative work). These analyses allowed an examination of the extent to which the five types of professional development were predictive of the use of standards-based curriculum and instruction and student achievement in science and mathematics.

The first two regressions were designed to examine the relationship between the different types of professional development and teachers' use of standards-based curriculum and instruction. (See tables 4 and 5.) For both science and mathematics teachers it was found that curriculum development and examining practice were significant predictors of teachers' use of standards-based curriculum and instruction. Immersion, curriculum implementation, and collaborative work were not found to be significant predictors in the model. For science teachers the model accounted for 35% of the variance of standards-based curriculum and instruction. For mathematics teachers the model accounted for 18.5% of the variance.

Table 3

Percent of Science and Mathematics Teachers Engaged in Different Types of Professional Development

The next two regressions were designed to examine the relationship between the type of professional development and student achievement. For science teachers, none of the different types of professional development were significantly related to student achievement. For mathematics teachers, only curriculum development was found to relate to student achievement; however, the relationship was negative (See Table 6). Immersion, curriculum implementation, examiningpractice, and collaborative work were not found to predict student achievement. The model accounted for 16% of the variance of student achievement.

Discussion

This study provides a unique look at the impact of professional development on student achievement. The results suggest that for both science and mathematics teachers participation in two types of professional development, namely, examiningpractice and curriculum development, are related to the use of standards-based instructional practice. Immersion, curriculum implementation, and collaborative work were not found to relate to teachers' use of standards-based instruction. These results appear to parallel previous research on this issue.

Table 4

Science Teachers' Duration of Different Types of Professional Development Regressed on Standards-Based Curriculum and Instruction

Table 5

Mathematics Teachers' Duration of Different Types of Professional Development Regressed on Standards-Based Curriculum and Instruction

Table 6

Mathematics Teachers' Duration of Different Types of Professional Development Regressed on Student Achievement.

In terms of examining practice, Desimore, Porter, Garet, Yoon, and Birman (2002) found that professional development focused on examining specific teaching practices can increase the use of those practices. For example, examining practice through the use of case discussions and teacher narratives has been found to alter teaching practices (Barnett, 1998; Schifter, 1996).

In addition, previous research suggests that the use of professional development focused on student thinking can be helpful for instruction designed to improve student understanding of mathematics concepts (Fennema et al., 1996). These findings and the results \of the present study appear logical. One would expect that having teachers examine their instructional practice would lead to changes in those practices.

The positive relationship between engaging in curriculum development and standards-based instruction is more difficult to explain, especially since curriculum implementation was not found to relate significantly to instruction. One explanation for this finding is that participation in curriculum development is selective, and only the more skilled teachers tended to engage in such activity. Additionally, professional development that is coherent with the daily environment of schools has been found to enhance practice (Garet, Porter, Desimone, Birman & Kwang, 2001). It has also been suggested that professional development should occur in the context of the daily work of teachers so that enhanced knowledge and skills can be validated in classrooms (Mewborn, 2003).

The teachers in the present study were likely to be involved in more local school-based curriculum development that was coherent with and that occurred within the context of their own schools and classrooms. For example, at one of the schools the teachers were involved in the development of curriculum materials to analyze their own local water resources. This situation may have produced more ownership on the part of teachers and better implementation and impact. The idea of ownership and coherence also helps to explain why curriculum implementation professional development did not relate to standards-based instruction. The teachers may have been expected to implement a curriculum they did not "own," which might not fit with their setting or students. Also, the teachers may have been reluctant to fully implement or may not have had the skills to implement as intended. This corresponds to ideas about the impact of empowerment in the literature (Louis, Marks & Kruse, 1996; Newmann & Associates, 1997).

Ferrini-Mundy (1997) pointed out that teachers need to be part of the process. Ultimately, teachers make the choice to implement or not. One way to get their ownership is to have them plan the changes. Tyack and Cuban ( 1995) favored a reform model that attempts to work from the "inside-out" by enlisting the skills of classroom teachers as key players in reform. Fullan and Miles ( 1992) also emphasized that ultimately change is always implemented by teachers in the classroom. Successful reform, according to Fullan (1991), requires that the change be viewed by the teachers as relevant to the needs of the school, that teachers be ready for change, and that teachers have the necessary resources to support change. Reform is local and ultimately controlled by teachers in the classroom; hence, ownership is essential.

Previous research also shows that ownership may be necessary for proper curriculum implementation. In a longitudinal study of curriculum implementation in high school science, Lawrenz, Huffman, and Lavoie (2001) reported that the schools where teachers instigated and pursued reform for their own sake were able to sustain the reform more than schools where the change was forced upon the teachers. More research, however, is needed to better understand the complex relationship between curricular implementation and instructional practice in mathematics and science.

Immersion and collaborative work were also not found to relate to standards-based instructional practices. Immersion is likely to have more of a long-term and amorphous effect, rather than relate directly to instruction or achievement. Many of the professional development activities that immerse teachers in the real-world of scientists or mathematicians strive for better understanding of the nature of science and mathematics and the complexity of the process, rather than skills or knowledge that can be directly applied to the classroom. The results might be apparent over time as teachers change their perspectives about science or mathematics, but one is unlikely to see an immediate impact on instruction or student test scores. Collaborative work is also highly recommended, but as with immersion it can be more amorphous and unfocused. In fact, Gerhart et al. (1999) found that professional development aimed merely at collaboration without a specific focus on topics such as student thinking, content, or curriculum was not as effective. Having teachers meet and discuss may be valuable, but it is not always focused on instructional practice and, as a result, may not strongly relate to instruction.

In terms of student achievement, only curriculum development for mathematics teachers was found to relate to student achievement. Mathematics teachers with students who have lower achievement were found to engage in more long-term curriculum development. One explanation is that these mathematics teachers may have engaged in more curriculum development because they had students who were low achieving and unresponsive to the traditional curriculum, and they strove to find and create something better. This is good news in that it appears the statewide system of professional development is reaching the teachers in most need.

On the other hand, teachers with students who have higher achievement do not appear to be engaged in as much long-term professional development. Their students are already doing well on achievement tests, and as a result, these teachers do not necessarily have as much motivation to engage in curriculum development or change what they are doing. Additionally, many other factors unrelated to the classroom and professional development affect student achievement. In this study curriculum development accounted for 16% of the variance of student achievement, suggesting that other factors account for more of student achievement. Previous research suggests that variables such as professional community, decentralized and expanded leadership, supportive school structures, and increased parental and community involvement can all significantly affect student achievement (Louis et al., 1996; Newmann, 1997; Stringfield, 1995). More research is needed on engaging teachers in curriculum development and its impact on student achievement.

Overall, the results of this study add to the literature regarding the impact of professional development. Although the different types ofprofessional development investigated in this study are all recommended as examples of high quality enhancement for teachers, it is informative to find that examining practice and curriculum development are most predictive of standards-based instruction. It is also important to note that this finding was apparent for both science and mathematics teachers. However, there appears to be only a weak relationship between these types of professional development and student achievement on state exams. While professional development is often touted as the key to science and mathematics education reform, it appears that professional development for individual teachers is not enough.

Sparks (2002) espoused a three-part premise related to professional learning. First, quality teaching can enhance student learning; second, a key factor in quality teaching is the professional learning of teachers and principals; and third, the structure and culture of the school environment are critical to enhancing the impact ofprofessional learning.

One could argue that the focus should be on schoolwide reform rather than on individual teachers. Using the school as the unit of change takes into consideration that student success is affected by more than the individual science or mathematics teacher. School structure and culture, school leadership, human resources and support, and the professional community are all important factors that need to be considered in efforts to improve student achievement. More research is needed to better understand how these various systemic reform factors interrelate with professional development and student achievement. Although such systemic reform is difficult to create, it may be one of the most important means of truly reforming science and mathematics for all students.

References

Ball, D. L., & Cohen, D. K. (1999). Developing practice, developing practioners: Toward a practicebased theory of professional education. In G. Sykes & L. Darling-Hammond (Eds.), Teaching as the learning profession: Handbook of policy and practice (pp. 3-32). San Francisco, CA: Jossey Bass.

Barnett, C. (1998). Mathematics teaching cases as a catalyst for informed strategic inquiry. Teaching and Teacher Education, 14(1), 81-93.

Cohen, D., & Hill, H. (1998). Instructional policy and classroom performance: The mathematics reform in California. Consortium for Policy Reform in Education (Issue Brief No. RB-23).

Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on Teachers' instruction: Results of a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81-112.

Education Commission of the States. (1997). Investing in teacher professional development: A look at 16 school districts. Denver, CO: Author.

Elmore, R. F. (1996). Getting to scale with good educational practice. Harvard Educational Review, 66(1), 1-26.

Fennema, E., Carpenter, T. P., Franke, M. L., Levi, L., Jacobs, V. R., &Empson, S. B. (1996). Alongitudinal study ofleaming to use children's thinking in mathematics instruction. Journal for Research in Mathematics Education, 27(4), 403-434.

Ferrini-Mundy, J. (1997). Common components and guiding principles for teacher enhancement programs. In G. W. Bright & S. N. Freilich (Eds.), Reflecting on our work: NSF teacher enhancement in K-6 mathematics. Washington, DC: The National Science Foundation.

Fullan, M. G. (1991). The new meaning of educational change. New York: Teachers College Press.

Fullan, M. G., & Miles, M. B. (1992). Getting reform right: What works and what \doesn't. Phi Delta Kappan, 73(10), 744-749.

Garet, M., Porter, A., Desimone, L., Birman, B., & Kwang, S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 35(4), 915-945.

Gerhart, M., Saxe, G. B., Seltzer, M., Schlackman, J., Ching, C. C., Nasir, N. N., Fall, R., Bennett, T., Rhine, S., & Slone, T. F. ( 1999). Opportunities to learn fractions in elementary mathematics classrooms. Journal for Research in Mathematics Education, 30(3), 286-315.

Horizon Research Incorporated. (1997). Local systemic change core evaluation data collection manual. Chapel Hill, NC: Author.

Lawrenz, F., Huffman, D., & Lavoie, B. (2001). The long-term effects of teacher enhancement. Washington, DC: National Science Foundation.

Louis, K. S., Marks, H. M., & Kruse, S. (1996). Teachers' professional community in restructuring schools. American Educational Research Journal, 33(4), 757-798.

Loucks-Horsley, S., Hewson, P. W., Love, N., & Stiles, K. E. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press.

Loucks-Horsley, S., & Matsumoto, C. (1999). Research on professional development for teachers of mathematics and science: The state of the scene. School Science and Mathematics, 99(5), 258- 271.

Mewborn, D. S. (2003). Teaching, teachers, knowledge, and their professional development. In J. Kilpatrick, W. G. Martin, & D. Schifter, (Eds.). A research companion to principals and standards for school mathematics (pp. 45-52). Reston, VA: The National Council of Teachers of Mathematics.

National Commission on Mathematics and Science Teaching. (2000). Before it 's too late: A report to the nation from the national commission on mathematics and science teachingfor the 21s' century. Washington, DC: Author.

National Commission on Teaching and America's Future. (1996). What matters most: Teaching for America 's future. New York: Author.

National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.

National Education Goals Panel. (1994). National education goals report: Building a nation of leaders (3rd ed.). Washington, DC: Government Printing Office.

National Research Council. (1996). National science ducation standards. Washington, DC: National Academy Press.

Newmann, F., & Associates (1997). Authentic achievement: Restructuring schools for intellectual quality. San Francisco: Jossey Bass.

Riley, R. (1998, September 15). The challenge for America: A high quality teacher in every classroom. Annual back to school address by U.S. secretary of Education, Richard W. Riley, National Press Club, Washington, DC.

Schifter, D. (1996). What's happening in math class? Envisioning new practices through teacher narratives (Vois. 1-2). New York: Teachers College Press.

Sparks, D. (2002). Designing powerful professional development for teachers and principals. Oxford, OH: National Staff Development Council.

Stringfield, S. (1995). Attempting to enhance students learning through innovative programs: The case for schools evolving into highreliability organizations. School Effectiveness and School Improvement, 6, 67-96.

Third International Mathematics and Science Study. (1996). A splintered vision: An investigation of US science and mathematics education. Boston, MA: Kluwer.

Tyack, D., & Cuban, L. (1995). Tinkering toward Utopia: A century of public school reform. Cambridge, MA: Harvard University Press.

Douglas Huffman and Kelli Thomas

University of Kansas

Frances Lawrenz

University of Minnesota

Editors 'Note: Correspondence concerning this article should be addressed to Douglas Huffman, University of Kansas, Department of Teaching and Leadership, 347 Joseph R. Pearson Hall, 1122 West Campus Road, Lawrence, KS 66045.

Electronic mail may be sent via Internet to [email protected]

Appendix

Items on the Teacher Survey

Curriculum & Instruction Scale

How often do students: (rarely or never, once a month, once a week, 2-3 times a week, daily)

1. Participate in student-led discussions.

2. Participate in discussions to deepen science/math understanding.

3. Make formal presentations to the class.

4. Read from a science/math textbook in class.

5. Read other science related material.

6. Work on solving real-world problems.

7. Share ideas or solve problems with each other in small groups.

8. Engage in hands-on science/math activities.

9. Follow prescribed steps in an activity or investigation.

10. Record, represent and/or analyze data.

11. Complete worksheets that emphasize mastery of essential skills.

12. Prepare written science/math reports of at least three pages.

13. Write reflections in a notebook or journal.

14. Describe what they know about a topic before it is taught.

15. Use community resources in the classroom.

16. Use calculators or computers to solve science/math problems.

17. Document and evaluate their own science/math work.

Copyright School Science and Mathematics Association, Incorporated Dec 2003