• E-mail
• Print
• Comment
• Font Size
• Digg
• del.icio.us
• Discuss article

Kentucky's Early Childhood Continuous Assessment and Accountability System: Local Decisions and State Supports

Posted on: Tuesday, 24 July 2007, 06:22 CDT

By Rous, Beth McCormick, Katherine; Gooden, Caroline; Townley, Kim F

Kentucky has developed an assessment and accountability system that provides outcome data for a set of state early childhood standards for children birth to 5 years of age that are aligned with early childhood outcomes designated by the Office of Special Education Programs (OSEP). Within this flexible and comprehensive state assessment system, early intervention and care providers, district preschool personnel, and childcare program staff select a criterion-referenced, curriculum-based assessment (CBA) tool to monitor children's progress. The system is designed so that data from these CBA tools can be used for three purposes: (a) to demonstrate child progress and inform instruction at the individualized intervention or classroom level; (b) to promote program improvement and plan program, district, and state supports through the aggregation of assessment data at the program, district, and state level; and (c) to report on child progress to evaluate the success of federal, state, or district programs and initiatives. Over the past 17 years, the Commonwealth of Kentucky has addressed the educational needs of its citizens through two reform efforts. In 1990, the Kentucky General Assembly passed the Kentucky Education Reform Act (KERA) and provided funding for sweeping changes in school practices to equalize educational opportunity for all students (Kentucky General Assembly, 1990). KERA completely restructured the curriculum of Kentucky's public schools and the finance and governance systems of public education. This reform also included the development of academic standards for kindergarten through 12th grade and a rigorous assessment system for accountability purposes. One component of KERA specifically targeted young children in the preschool years of early childhood.

Through KERA, the public preschool program was designed to provide services for at-risk 4-year-old children, defined as children whose families' incomes are at or below 150% of the poverty level, and for all 3- and 4year-old children who meet state guidelines for special education and related services, regardless of income. Districts may choose to enroll other 3- and 4-year-old children on a tuition basis or through the use of local funds. KERA also requires that districts work with existing community early care and education programs to avoid duplicating programs and services or supplanting federal funds. Districts must sign a full utilization agreement with Head Start to serve as many 4-year-old children as possible and to maximize federal Head Start funds.

In most public preschool settings, programs provide inclusive environments for the provision of special education and related services to eligible preschool children. Programs are designed to be developmentally appropriate for all young children and are defined by state law to focus on the child's physical, intellectual, social, and emotional development. In response to the needs of school districts to staff these new programs, a new birth-to-5 interdisciplinary early childhood education teaching certificate was also developed (see McCormick & Brennan, 2001).

In 2000, Kentucky implemented a second and even more comprehensive early childhood initiative that was viewed as the "next logical step in Kentucky's educational improvement efforts" (Governor's Early Childhood Task Force, 1999, p. 2). In the same way that KERA addressed services for young children in Kentucky public schools, this initiative focused on services to young children and their families across childcare and other service systems. Kentucky's KIDS (Kentucky Invests in Developing Success) NOW Initiative (HB 706) was designed to establish and coordinate supports and services needed within the commonwealth to ensure that "all young children in Kentucky are healthy and safe, possess the foundation that will enable school and personal success, and live in strong families that are supported and strengthened within their community" (Governor's Early Childhood Task Force, 1999, p. 2). The rationale for the implementation of KIDS NOW was the lack of a comprehensive state plan to support quality early childhood experiences in all of the environments in which young children spend time. The initiative focuses on outcomes in five areas: (a) ensuring maternal and child health; (b) supporting families; (c) enhancing early care and education; (d) establishing a support structure; and (e) improving programs. This article describes (a) the development and implementation of these comprehensive initiatives in Kentucky to increase the quality of opportunities for all young children and (b) the development of a system of accountability based on improved assessment and curriculum practices that result in improved outcomes for all young children.

STATE/FEDERAL INITIATIVES

The need for improved accountability in the field of education has followed multiple reform efforts over the years, which have had as a common objective increased student achievement and systemic capacity to demonstrate evidence of this achievement (McLaughlin, Henderson, & Rhim, 1998). The most current wave of accountability demands in early childhood has been greatly influenced by the reauthorization of Title 1, through the No Child Left Behind (NCLB) Act of 2001, and President Bush's Good Start Grow Smart Early Childhood Initiative (2002). Early childhood education programs across the country have also been affected by other federal initiatives, specifically the Governmental Performance Reporting Act (GPRA) and the Program Assessment Rating Tool (PART; Harbin, Rous, & McLean, 2005). Both GPRA and PART required all federal programs (e.g., Head Start, childcare, and programs for children with disabilities) to report data on progress toward meeting the goals of the program, which in turn are used to help determine federal budget allocations.

While accountability efforts have been moving forward at the national level, states have been engaged in parallel and complementary accountability efforts through the development of mechanisms to document the impact of the early childhood services provided in their state and the outcomes for young children resulting from these services. For many states, the development of a set of stateidentified early childhood standards has been a critical part of this process. At the current time, 43 states have developed early childhood standards (Neuman & Roskos, 2005). In some states, systems have been designed in direct response to federal or state initiatives (e.g., Good Start Grow Smart, National Head Start Reporting System, School Readiness) and often are focused on specific content (e.g., numeracy and literacy; Scott-Little & Kagan, 2004) or specific populations (e.g., Head Start; Lund, Rous, Purvis, & Moffett, 2003). Common among all these efforts is the intent that standards articulate what children should know and be able to do.

For young children with special needs who are participating in Part C and section 619 programs, the need for greater accountability was also prompted by poor evaluation data reported through GPRA and PART. In response, the Office of Special Education Programs (OSEP) established three child outcomes by which states will measure progress for children with disabilities, with entry data provided to OSEP in the Annual Performance Report of February 1, 2007. These three outcomes, developed by OSEP with assistance from the National Early Childhood Outcomes Center (ECO), are designed to provide national data on the percentage of young children who are participating in early intervention and preschool services and who demonstrate improved (a) positive socialemotional skills, (b) acquisition and use of knowledge and skills, and (c) use of appropriate behaviors to meet their needs.

Several states, and the commonwealth of Kentucky, have taken a proactive and comprehensive approach in meeting the new accountability requirements and are designing standards and accountability systems that encompass all content areas, all children, and all programs. In Kentucky it is anticipated that by using a universal accountability system, the commonwealth will be capable of responding to the needs of multiple constituents in support of a high-quality, comprehensive system of interagency services and programs for all young children, including those with disabilities.

KENTUCKY'S KIDS NOW INITIATIVE: BUILDING A STRONG FOUNDATION FOR SCHOOL SUCCESS

Recognizing that increasing national and state attention has been paid to the early childhood years as the foundation of children's academic success and that positive child outcomes, specifically behavior and early development, have been correlated with lower child/staff ratios, smaller group sizes, and higher levels of caregiver education and training (Burchinal, Roberts, Nabors, & Bryant, 1996; Clarke-Stewart, Gruber, & Fitzgerald, 1994; Dunn, 1993; Phillips, Howes, & Whitebook, 1991; Ruopp, Travers, Glantz, & Coelen, 1979), a task force of early childhood educators, advocates, and key stakeholders began the design of accountability processes that included four levels, referred to as the Four Key Dimensions for Success (Rous & Townley, 2004). The four key dimensions address standards at the state, personnel, environmental, and child levels. To address the state-level dimension, Kentucky participated in a 17- state School Readiness Initiative (http: //www.gettingready.org/ matriarch/) through which indicators of child well-being were established to help measure overall success of the state in addressing the needs of young children and families. The second key dimension for success focused on early care and education personnel. Kentucky's legislature authorized the development and provision of a seamless system of professional development in early childhood, as well as incentives that would attract and retain providers (see Figure 1). This resulted in a Professional Development (PD) Framework that consists of five components: (a) professional core content; (b) provider, director, and trainer credentials; (c) a formal training framework; (d) an articulation plan across credentials and degrees; and (e) college and noncollege scholarships. A more detailed description of the Kentucky PD Framework is provided by Rous, Grove, and Townley (2007).

At the program level (the third key dimension for success), environmental standards were developed and outlined through the ST*RS for KIDS NOW Quality Rating System for Child Care Centers and Homes and the Quality Self-Study (Newberry & Howard, 2005). These standards are focused on increasing the level of quality in programs that provide early care and education for young children and thus support positive outcomes for children and families. An evaluation of KIDS NOW is currently under way to determine the impact of various aspects of the KIDS NOW Initiative on improved program quality and child outcomes (more information about the KIDS NOW Evaluation is available at http:// www.kidsnow.ky.gov).

EARLY CHILDHOOD STANDARDS

The final key dimension for success-and the focus of this article- is at the child level and includes the Kentucky Early Childhood Standards (Kentucky Standards). The use of standards as an important component of early childhood reform and accountability efforts was endorsed in a 2002 joint position statement by the National Association for the Education of Young Children (NAEYC) and the National Association of Early Childhood Specialists in State Departments of Education (NAECSSDE). In Kentucky, the early childhood standards were developed for children ages birth to 5 years and include a set of standards, benchmarks, developmental continua, and examples. Kentucky standards are defined as general statements that represent the information that a child should know and the skills that he or she should be able to do during the early childhood years, a definition congruent with those used by other early childhood researchers (see Bodrova, Leong, & Shore, 2004). Kentucky standards include both content (e.g., the what) and performance (e.g., the how) standards.

Development of the Kentucky Standards

To be useful for teachers and interventionists in improving instruction and individualized intervention, standards must be broken into smaller behaviors or multiskill sets as specific statements that can be demonstrated and measured (Bodrova et al., 2004). A developmental continuum is provided for each benchmark and represents a predictable sequence of accomplishments that describes the progressive levels of performance in the order in which they emerge in most children, based on current research (Burns et al., 2003; Kendall, 2003). Example behaviors represent observable "samples" of what children might do as they demonstrate accomplishments at each level of the developmental continuum, although they are not intended to be a definitive list of how a child might demonstrate a specific accomplishment or an exhaustive inventory of those skills.

This tiered system (standard, benchmark, developmental continuum, and example behavior) has three distinct advantages. First, it provides a level of specificity for describing child status across diverse ability levels. second, it is designed to provide support for personnel at various proficiency levels (e.g., from entry into the field to master level) as they develop curriculum, intervention programs, and individualized plans (e.g., Individualized Education Programs [IEPs]/Individualized Family Service Plans [IFSPs]) for children who are participating in their programs. Third, the system supports a range of programs across services delivery models and structures (e.g., center or home based, preventative or intervention) and provides a mechanism for shared expectations across settings.

The Kentucky Standards were developed over a 2-year period by a workgroup of early childhood experts and stakeholders who represented each of the early childhood programs in the state (e.g., Early Intervention, public preschool, Head Start, institutes of higher education, and training and professional development providers). Once developed, the standards underwent a three-stage field validation and review process: (a) state agency, (b) in-state expert, and (c) national expert. (A more detailed description of the process is provided in the Kentucky Standards document, which is available at http://www.kidsnow.ky.gov)

The Kentucky Standards include universal expectations for all children ages birth to 5 years and are divided into two sets: for ages birth to 3 and for ages 3 through 4 years. The birth-to-3 standards include five developmental domains: creative expression, communication, cognitive, motor, and social-emotional. Approaches to learning and cultural appropriateness are embedded throughout the standards. By design, the developmental continuum for each domain of the birth-to-3 standards is not matched with particular ages of infants and toddlers.

The birth-to-3 standards are aligned with the 3- and 4-year-old standards, which are organized by content areas: arts and humanities, English/language arts (early literacy), health education (health/mental wellness), mathematics, science, physical education (gross and fine motor skills), and social studies (see Figure 2). These content areas are consistent with and aligned to the entry- level skills in the K-12 Program of Studies (i.e., school-age standards). In addition, these standards are aligned with the Head Start outcomes and the OSEP early childhood outcomes.

The development of the Kentucky Standards was based on several key principles. First is the recognition that social-emotional experiences and relationships form the foundation for child development (Chazan-Cohen, Jerald, & Stark, 2001) and serve as predictors of later success (Kontos & Wilcox-Herzog, 1997; NAEYC & NAECS/SDE, 2003; Peisner-Feinberg et al., 2001). Second is acknowledgment that early care and education programs must use research-based, recommended practices and that these practices should be based on knowledge about (a) child development and learning; (b) the strengths, needs, and interests of each particular child to allow for adaptation for individual variation; and (c) the social and cultural contexts in which a child lives, to ensure that experiences are meaningful, relevant, and respectful (NAEYC, 1997, p. 5). Third, the interaction and influence among developmental domains must be a consideration in addressing program and child needs and outcomes, given that young children's development is strongly interconnected (Kendall, 2003). Fourth, the individual developmental needs of children must be addressed and the child recognized as a unique person with an individual personality, learning style, and experiential background (NAEYC, 1997, p. 6). Fifth, the standards were designed to help teachers and providers understand the ecological nature of early childhood (Bronfenbrenner, 1993) and address the cultural and language differences among children and families (NAEYC, 1997). Finally, the standards are designed with an understanding that the quality of early care and education programs affects short- and long-term outcomes for children (Howes, 1997).

Within early childhood settings, instruction, curriculum, and assessment are appropriately joined to promote each child's progress. The National Institute for Early Education Research (NIEER; Bodrova et al., 2004) reported that child outcomes are improved when teachers report that they implemented policies of standards-based reform. An informal survey conducted by the Kentucky Department of Education (KDE), as well as comments from other sources who interact with early childhood personnel (coordinators and teachers), indicated that early childhood program staff are using the Kentucky Standards to align school districts' curricula, create curriculum maps and standards-based units of study, evaluate learning centers and classroom activities, and individualize instruction and IEP development (N. Newberry, personal communication, September 10, 2006).

Continuous Assessment System

The next step in building the Kentucky accountability system was the development of the Continuous Assessment Guide (Rous & Townley, 2004). While the Kentucky Standards were designed to help ensure quality early care and education services by providing a common language through which program staff can express expectations for young children and the Early Childhood Quality Self-Study (Newberry & Howard, 2005) was designed to help programs evaluate their services and determine areas of strengths and need to better help children meet the early childhood standards, the Continuous Assessment Guide provided a common process for observing and recording the progress of children on those standards. When used in concert, the standards and assessment system provide necessary information to help early care and education program personnel document both the results and benefits of their programs and interventions.

The Continuous Assessment Guide includes recommendations for assessment tools across three levels: screening, diagnostic (to determine eligibility or need for further services), and classroom/ instructional (curriculumbased or embedded). Program evaluation is also provided in the guide and consists of recommended procedures rather than tools. Like the other levels of the accountability system, the Continuous Assessment Guide was designed for all children and all programs (public and private) in both classroom and home-based settings. A statewide workgroup of early childhood academicians and practitioners was recruited to identify the specific criteria for each type of assessment tool (i.e., screening, diagnostic, and classroom/instructional) and to determine which assessment tools met the criteria for recommendation in the Continuous Assessment Guide. A consensus model was used for the selection of the criteria for each of the three types of assessment tools as well as the determination of which tools met these criteria.

The focus of the discussion included in this article is assessment tools that are classified as classroom/ instructional in the Continuous Assessment Guide and are most commonly referred to as criterion-referenced or curriculum-based assessment tools (CBA). For these tools, 12 criteria were determined by the workgroup as critical for selection: (a) demonstrates technical adequacy, (b) focuses on functional goals and outcomes, (c) includes multiple domains, (d) is appropriate for diverse learners, (e) provides opportunities for a variety of methods of data collection, (f) addresses behaviors and skills across the early childhood age range, (g) includes families in the process, (h) is usable by program staff with training, (i) yields both quantitative and qualitative data, (j) is currently used in Kentucky, (k) is not cost prohibitive, and (1) does not require prohibitive time to administer. A more thorough description of the criteria for each assessment type is provided in the Continuous Assessment Guide (see http://www.kidsnow.ky.gov).

Throughout the process, teachers and other early childhood personnel from across the commonwealth sent recommendations of specific assessment instruments to the workgroup for review. The workgroup determined that 12 CBA tools met their criteria. Once appropriate tools were identified, an expert panel of early childhood professionals worked in cooperation with the publishers of each tool to align the items on each of the assessments with the Kentucky Standards. This process was designed to help programs make informed decisions about the assessment tools they would use as they developed their local early childhood continuous assessment system. Selection decisions were made at the local district or program level to determine the most appropriate assessment tool for specific programs and children. These selection decisions were made based on local criteria (i.e., resources, past training, and preference for reporting). A more detailed discussion of this process is included later in this article.

The 12 recommended classroom/instructional assessment tools vary on multiple dimensions, such as the specificity of the items included in the assessment and the age ranges covered by the assessment. Yet the tools have two important features in common. First, they are all criterion referenced and share properties related to assessments that can be identified as curriculum based. Salvia and Ysseldyke (2007) defined curriculum-based tools as "assessment methodologies that are used to collect and evaluate student achievement data in order to monitor student progress" (p. 636). Over time these tools have come to be used most often as assessments for monitoring student progress through the curriculum. Second, all of the tools (with varying levels of specificity) address all five domains of learning in the birth-to-3 standards and all seven content areas in the 3- and 4-year-old standards. The recommended tools are included in Table 1.

The Continuous Assessment Guide was designed to support a continuous assessment process that includes the use of criterion- referenced CBA tools that utilize multiple sources of information (e.g., families, paraeducators, related service personnel) and multiple methods of data collection (e.g., interview, portfolio, observations). This model allows teachers to continually update information on children and change their intervention or instruction in response to assessment outcomes (Deno, 2003). Within this system, assessment has two purposes: to describe child status and to inform instruction or intervention. In this way, provider and teacher instruction and intervention are informed by ongoing assessment for maximal progress by all children, including those with disabilities and special needs. This system also supports a standardsbased curriculum and intervention model that supports families, early intervention providers, teachers, and related personnel in their use of recommended practice guidelines for continuous, authentic assessment while providing child outcome information to families, teachers, interventionists, and local, state, and federal stakeholders on a common set of state standards.

In summary, the Kentucky Standards and continuous assessment system have been developed to be responsive to the need for programs to gather ongoing data on children's progress through the selection and use of criterion-referenced CBA tools that (a) align with program goals and (b) provide authentic, outcome data on all children, both with and without disabilities, enrolled in the program. The design of the system reduces the need for programs to alter their assessment tools and procedures in response to the ever- evolving and increasing requirements for state and national accountability and outcome data across agencies and programs.

Validating the Link Between Assessment Items and Child Outcome Data

The Kentucky accountability system examines child progress on state standards and OSEP child outcomes through progress on a set of 12 curriculum-based instruments. The system does not link assessment items directly to the OSEP outcomes but rather to Kentucky benchmarks and standards and then to the OSEP outcomes. This approach to measurement of OSEP outcomes reports individual child assessment data in two separate and equally important directions: (a) individual child data to Kentucky benchmarks and standards and (b) individual child data to benchmarks and standards and then to OSEP outcomes (see Figure 3).

To design an accountability process that allowed programs to choose from a set of recommended assessment tools and to report data from any one of these assessment tools on a common set of standards and outcomes, four specific validation activities were designed and implemented. This was done in tandem with data from a pilot study funded through a General Supervision Enhancement Grant (GSEG) from OSEP and data from the first cohort (Phase 1) of local preschool districts reporting as part of the State Performance Plan (SPP; KDE, 2005) supported with funding from KDE.

The first step in the validation process was to verify the earlier work completed by KDE early childhood experts and the publishers of each tool to cross-reference test items from the recommended assessment tools with the Kentucky Standards. The cross- referenced work is referred to as crosswalks, and is published in the Continuous Assessment Guide. This process was designed to determine the degree to which items across assessment tools consistently measure Kentucky standards using the related benchmarks. Kentucky's standards were then aligned with the OSEP child outcomes based on the SPP.

This work is being conducted by a panel of five state early childhood experts who follow a specific protocol and guidelines approved by the GSEG planning team, which included university and state representatives. These experts have experience at both the state and program levels, and they have content knowledge and advanced degrees in early childhood research, child development, developmental psychology, early childhood, and early childhood special education. The team met regularly for more than a year to begin the process for those tools used by 23 public preschool programs (the Phase 1 sites) and continues to meet to review additional approved tools used by early intervention providers.

The panel first reviewed existing publisher-validated crosswalks that aligned assessment items from each tool to the Kentucky benchmarks, which define or operationalize the broader Kentucky standards. Once benchmark-level crosswalks were completed, standard- level crosswalks were compiled. Last, the Kentucky benchmarks and standards that were determined to best measure the OSEP child outcomes were compiled for the third-level crosswalk, the measurement of progress toward the OSEP outcomes. These three levels of crosswalks are being refined by panel review and statistical analyses at each level (benchmark, standard, and OSEP outcome). The crosswalk work, as begun by KDE staff and publishers and then refined by the panel, will continue to be validated and refined as additional data are collected.

In addition to the alignment work, the early childhood panel also began the process of determining age bands for specific items on the assessment tools, using 6-month increments for 3- and 4-year-olds and 3-month increments for birth to 3-year-olds. This work was completed first on a theoretical level, using publisher information when available and other developmental guidance as recommended (e.g., Cohen & Gross, 1979). This process is similar to one described by Macy, Bricker, and Squires (2005) in determining the validity and reliability of using a curriculum-based assessment to determine Part C eligibility. This process is also in keeping with recommendations by other researchers, such as Lewis and Haug (2005), who reported that the outcome of setting performance standards would be the generation of a set of cut scores that operationalize a set of performance levels. As more data are collected across the commonwealth, statistical analyses will further refine the current alignment of items to benchmarks and standards. These statistical analyses are the second part of the validation process for Kentucky's system of assessment and accountability. Kentucky has begun ongoing analyses of child assessment data with Kentucky's standards, benchmarks, and the OSEP child outcomes for the assessment tools chosen by three pilot sites: The Carolina Curriculum for Infants and Toddlers with Special Needs (CCITSN; Johnson-Martin, Attermeier, & Hacker, 2004), The Creative Curriculum Developmental Continuum for Preschool (Dodge, Colker, & Heroman, 2001), and the High/Scope Child Observation Record (COR; High/Scope Educational Research Foundation, 2003). The preliminary analyses indicated moderate to strong correlations between assessment items and Kentucky benchmarks for each of these curriculum tools. As data are added this calendar year from additional preschool districts and children who participate in Part C services, additional statistical analyses will be conducted to include techniques such as factor (where sample size is sufficient) and additional correlational analyses. Data will be analyzed for each of the crosswalk levels (benchmark, standard, and OSEP child outcome) to determine the most reliable and informative way to assess children's progress for local, state, and federal OSEP reporting purposes.

The third part of the validation process included activities that examined the relationship between (a) teacher perceptions of children's progress toward the standards and benchmarks and (b) OSEP child outcomes and the assessment tool results. A series of rating and ranking activities were conducted with pilot site personnel in order to measure staff perceptions of student progress toward the standards. Staff rated the performance of their students according to progress toward the standards and benchmarks within 2 weeks of completing an assessment using the district-selected tool (i.e., COR; Assessment, Evaluation, and Programming System [AEPS; Bricker, Capt, & Pretti-Frontczak, 2002]; Creative Curriculum [Dodge, Colker, & Heroman, 2001]).

To examine staff perceptions of children's progress toward OSEP child outcomes, staff members ranked their students in order from highest functioning to lowest functioning on each of the three OSEP child outcomes, using the ECO Center guidelines for descriptions of each outcome. Again, this activity was conducted within 2 weeks of the completion of a data checkpoint. For both activities, staff members were instructed to rate and rank their students based on their professional judgment and not on the results of their assessment tool. The teacher perception of progress data will then be compared to the assessment results for each tool to determine the congruence of teacher perception and assessment tool measurement of child status and progress.

The fourth phase of the validation process is a series of concurrent validity studies of approved tools used by the pilot sites. The criterion tool, the Battelle Developmental Inventory-2 (BDI-2; Newborg, 2005), was used for a sample of 38 children in the preschool sites. The BDI-2 was chosen because of its frequent use in the field as well as its use in other studies of concurrent validity of early childhood instruments (Feldman, Haley, & Coryell, 1990). The BDI-2 was administered by certified examiners within a month of administration of the curriculum-based tool. The results will be compared to determine the relationship between outcomes on a normreferenced assessment and outcomes from a subset of the recommended curriculum-based measures.

The validation approach used in Kentucky is similar to the approach used by other special education researchers in the alignment of alternate assessment of content with academic curricula. For example, Browder and her colleagues (2004) used a methodology reported by Crocker and Algina (1986). The procedures used were "(a) defining the performance domains of interests [standards]; (b) selecting a panel of qualified experts; (c) providing a structure framework for the process of matching items to the performance domain; and (e) collecting and analyzing the responses" (Browder et al., 2004, p. 213). However, in the Kentucky early childhood approach, a fifth step was added that included the empirical validation of the outcomes from the first four steps. In another case, Flowers, Browder, and Ahlgrim-Delzell (2006) described a similar alignment process proposed by Webb (1997), which combined the qualitative judgments of experts with quantified coding and resulted in a set of statistics to describe the degree of alignment of intersection between the content in the state standard and the content in the state assessment.

The Continuous Assessment Guide continues to evolve based on implementation and evaluation outcomes. Refinement is based on outcomes from the GSEG pilot study and KDE Phase 1. The pilot work was conducted with three early childhood programs: one childcare program serving children from birth to 5 years, one public preschool program serving children ages 3 to 5 years, and two early intervention providers. Preliminary data from the pilot study (n = 456) yielded positive correlations between Kentucky Standards and subscales of the approved tools used by the pilot sites. The Phase 1 work has initially involved public preschool programs in 23 school districts, including the two largest metropolitan areas in the state, for a sample of 7,022 children (1,653 of whom had IEPs, according to state records).

The pilot work was designed to collect data on the extent to which young children, including those with disabilities, met the Kentucky Standards and the OSEP child outcomes. This work consisted of four major activities:

1. gathering information on the current status of pilot programs/ providers in using the Kentucky Standards and Continuous Assessment Guide and determining the processes used to measure child and program outcomes;

2. providing assistance to pilot programs in the development of a comprehensive, continuous assessment system that will provide data on child and program outcomes;

3. providing assistance to programs in using assessment data to document progress for children and to monitor and document program and instructional changes; and

4. gathering information from programs about the assessment process that can be used to provide assistance and support to other programs across the state.

The Phase 1 work consisted of three major activities: (a) collaboration with school districts to determine which assessment tools the district would select for the collection of data on children enrolled in their programs, (b) collaboration with KDE to design a data platform that would allow for data entry of child outcomes across multiple programs and multiple assessment tools, and (c) collaboration with KDE to merge data from an extant state information system into the newly created data platform so that assessment information could be linked to specific children, teachers, and schools.

Districts participating in the Phase 1 study selected six assessment tools (from the set of 12): (a) The Creative Curriculum for Preschool, (b) COR, (c) AEPS, (d) Work Sampling System (WSS; Dichtelmiller, Jablon, Marsden, & Meisels, 2001), (e) Learning Accomplishment Profile-3 (LAP-3; Sanford, Zelman, Hardin, & Peisner- Feinberg, 2004), and (f) Brigance Early Inventory of Development11 (EID-II; Brigance, 2004). These six tools were then used in the initial data collection and analysis.

Designing a Universal Data Platform

The next and final step in the design of the system was the development of a Web-based platform, the Kentucky Early Childhood Data System (KEDS), that allowed programs to send child assessment data to be analyzed and aggregated across sites even when data were generated using the six assessment tools chosen by the 23 districts participating in Phase 1. The platform allows for the merging of two types of data for statistical comparisons to respond to state and federal data reporting requirements. Demographic data are collected from KDE's student information system (STI) and are downloaded onto the KEDS site. The demographic data include student identifying numbers, age, IEP considerations, gender, ethnicity, enrollment and exit dates, and economic indicators (free or reduced lunch). Preschool coordinators, teachers, or providers log on to the KEDS Web site using a self-generated access number and verify the downloaded child information data for the children in their class (or district, in the case of preschool coordinators). To protect the confidentiality of students in districts, preschool staff members are only allowed to verify information about the students in their class, while preschool coordinators have access to data for all students in their district. Staff members also indicate which assessment tool they use for each student and the mode of data entry (online, diskette, or paper). In addition, teachers enter data about the nature and amount of training they received on the assessment tool used. This information will be used for later reliability analysis, in-service training, and professional development planning.

The second type of data collected are actual child assessment data for each student. As with the demographic data, preschool staff members enter data only for their own students. Designated administrators and coordinators are able to view, enter, and monitor data for all children in their district. The collection of child assessment data for the KEDS platform involves three steps. First, the publisher of each assessment tool was contacted to determine the current methods for gathering data (e.g., online, diskette, paper). For assessment tools that involved a paper-protocol data collection, negotiations with the publisher on designing a data entry system as part of the Web-based platform were conducted and permissions were secured. Preschool staff members then entered data as recorded on their completed paper protocols onto a Web-based Excel 2003 (Microsoft(R)) spreadsheet structured much like the protocol. The Excel sheets were then uploaded and saved for analyses. For online and diskette systems, processes for accessing information from programs were also negotiated and permission secured from the publishers. KEDS technology staff are given administrator status for online assessment systems for each district, per the designation of each preschool district. KEDS staff upload the district assessment data with the assessment data for each child. No additional step for the transfer of online data is required by programs. Programs that use diskette systems export their saved data to the KEDS system from their diskette, according to permissions and directions for each tool. Specific directions for the transfer of data for all modalities (online, diskette, paper) are included on the KEDS Web site. Analyses of KEDS Data and Reporting

The KEDS system will eventually allow for the correlation of data from the 12 recommended assessment tools to the state's benchmarks and standards and OSEP outcomes (see Figure 3). For the analyses of the data from the Phase 1 districts, once data are entered into KEDS, they are exported to Microsoft Excel 2003 to obtain a comparable data set with the data analysis software and then are exported into SPSS version 14, in which codebooks for each variable have been developed. The data are then reviewed and cleaned using the measures of central tendency as well as skewness and kurtosis for each item on the six tools (in Phase 1). Any outliers are deleted from the analysis. Each child's assessment protocol is checked for incomplete data, and if more than 30% of the scores are missing, the child's data are deleted from the analysis. No plans are currently in place for inputting missing data. Data are then merged according to students' identification numbers, schools, and districts. When the data are merged in this manner, all assessment variables remain attached with the student variables and can be analyzed across multiple domains. Preliminary data analyses include determining children's age functioning for reporting OSEP outcomes.

The crosswalk documents are used as a guide to identify, by benchmark, the key items and their level within each assessment tool. Once key items are identified and grouped according to assessment tool, analyses are begun for reporting percentages for each of the three OSEP outcomes. Each child's score on the key items is computed into a percentage correct score for each of the assessment tools. Once the percentage correct per tool is calculated, the overall percentage correct for all tools is calculated by taking the mean percentage for each OSEP outcome. Data analysis for the OSEP outcomes is the first of many steps in analyzing the KEDS data. The next step in the KEDS data analysis will be the merging of data across schools and districts to validate each of the curriculum-based assessments.

The benchmarks, standards, and approved assessment tools are designed for use with all children ages birth through 4 years. Through the use of a variety of assessment tools and authentic measurement procedures that can be used to assess children at multiple developmental levels, children with disabilities are assessed and provided services with their age-appropriate peers within the same assessment and accountability system and services. Thus, the system facilitates the inclusion of children with disabilities within the most developmentally appropriate practice model.

DISCUSSION AND FUTURE PLANS

The long-term goal of the Kentucky assessment and accountability system is to provide data on the degree to which all young children who are participating in early care and education programs in the commonwealth are progressing on the early childhood standards. Beginning in the fall of 2007, the KEDS systems will allow for the addition of 22 school districts to the current 23 participating districts, with all districts joining the system by 2011. Also beginning in spring of 2007, the Kentucky early intervention Part C system will begin using KEDS to provide data on the OSEP outcomes. Additional plans are being discussed for the inclusion of other early care and education programs in the commonwealth as resources and provisions are made available.

It is anticipated that the KEDS system will facilitate improved instructional strategies for children with disabilities because teachers will participate in a continuous assessment process. They will receive aggregate data at the district or program level on the overall progress of children in meeting the standards and/or OSEP outcomes. Teachers input assessment data on children in the manner locally determined by the program and assessment availability, either online or with software. Once childlevel data are entered into the system, a variety of factors can be analyzed, including therapeutic services, dual placements, home or center placements, demographic characteristics, and more. The aggregation of data at the state level across programs provides opportunities to refine each dimension of the system (personnel, program, and child) and to inform multiple audiences and constituents. Statistical techniques requiring large sample sizes can be employed on an aggregate level to answer accountability questions posed by stakeholders at all levels of the early childhood ecology-family, teacher/provider, school/program, district, agency, state, and federal.

Multiple challenges are inherent in the type of system Kentucky has developed. Three are discussed here. First, and common to any assessment system, is the training and technical assistance necessary to obtain valid and reliable child performance outcomes from assessment tools. This challenge is especially problematic in Kentucky's system, which supports local autonomy in the selection of an assessment tool from among a fairly large array of tools with diverse psychometric properties and attributes. To support the selection of these tools, three training and technical assistance resources were made available to the early childhood community. First, a series of targeted workshops were conducted to provide introductory information about the 12 assessment tools. Second, workshops on specific assessment tools were made available to district personnel during the summer prior to implementation. Finally, a list of trainers from across the state who represented various training organizations (e.g., Regional Training Centers, colleges and universities, etc.) was compiled, and these trainers are available to provide on-site and regional training on the 12 assessment tools.

However, as stated earlier, conversations with some district- level preschool coordinators have suggested that personnel are selecting tools using a variety of criteria. Some individuals are choosing tools that their teachers/ providers currently use or have used in the past. Others are selecting tools based on past training or the availability of future training. Still others have suggested that the work of state staff and administrators (as described in this article) in the study of the attributes of the instruments will help them to identify the tools that provide the best match for program goals and staff capacity. The availability of ongoing technical assistance and support for online reporting, retrieval, and record keeping are also critical factors. Other variables important in the selection are cost, personnel expertise, usefulness in an interdisciplinary teaming model, goodness of fit with population characteristics, and personal preference.

A second challenge is the time-intensive nature of the development of a data platform that can be responsive to the individualized reporting needs of multiple state programs that serve young children within the state. We are fortunate in Kentucky to have the philosophical and financial support of section 619 and Part C programs to make the platform a reality. A third challenge is the ability to maintain and monitor the reliability of assessment procedures and data entry at the teacher or provider level. Continued concurrent validity studies and results of teacher/ provider assessment training data must guide the provision of training efforts. Although these challenges are important, they are procedural and can be overcome in time.

Two additional challenges are more systemic and include the use of curriculum-based tools within an accountability system and the linkage of these tools to a set of state-specific standards. Assessment presents one of the greatest challenges for states as they introduce preschool (and early intervention) outcome standards (Bodrova et al., 2004). Kentucky is not unique in identifying these challenges in measurement. A recent meta-analysis of child outcome measures in the study of childcare quality (Zaslow et al., 2006) suggested that measurement concerns are common in research in early care and education. For example, in an examination of 52 studies of socioemotional development, Zaslow and colleagues found at least one methodological problem for more than 70% of the studies; most often this concern was the failure of the researchers to provide sufficient information on the validity and reliability of the tools. The technical adequacy of measurement tools is a pervasive issue of concern in early childhood research and accountability efforts. However, a number of studies are currently being conducted by researchers in Head Start, childcare, and early childhood special education, as well as by the publishers themselves, to address issues of technical adequacy raised by the field (Zaslow et al., 2006). Never before have curriculum-based tools been used for high- stakes decisions in early childhood. Their use has historically been to document progress within programs and to inform instruction and intervention. There are few empirical studies that investigated their measurement properties for young children, although data are available that support the reliability, content, and criterion- related validity of CBA for older children (Deno, Marston, & Tindal, 1986; Shinn & Marston, 1985; Shinn, Tindal, Spira, & Marston, 1987; Shinn, Ysseldyke, Deno, & Tindal, 1986). It is assumed that the developmental sequences used in these tools are linear, exist for all children, are reasonably sensitive across ability levels, and can be used with confidence for children with disabilities or diverse behavioral repertoires; however, there are little empirical data to support these assumptions. Similar assumptions have been made for the Kentucky Standards.

Although the early childhood standards demonstrate social validity, empirical validity has not yet been established. Early childhood researchers suggest that in many areas of development there is insufficient evidence to identify reasonable expectations for young children (Bodrova et al., 2004). This is especially true for young children with disabilities and special needs. Furthermore, the relationship of the example behaviors, development continuum, benchmarks, and standards has also not yet been thoroughly investigated. For example, it is not known if certain benchmarks are more predictive of success on a standard than others. In addition, benchmarks are not necessarily equivalent; the scaling is not equal interval. Therefore, the data necessary for state and federal accountability initiatives will also be used to answer some of these important questions about the measurement attributes of this set of standards. Finally, "standards only work if the skills and concepts children are expected to master are the same skills and concepts they are taught in the classroom and the same skills and concepts that are measured by assessments" (Bodrova et al., 2004, p. 7).

Despite these challenges, the Kentucky continuous assessment and accountability system has several strengths. First, the system was developed through consensus of stakeholders from multiple early childhood agencies and programs. The system has been carefully developed so that it can be responsive to ongoing evaluation and feedback. Evaluation and feedback mechanisms have been institutionalized and well disseminated. In addition, the early childhood community in Kentucky is a coordinated system of agencies and providers. Local Community Early Childhood Councils (CECC), made up of personnel from various early childhood agencies and programs, such as Head Start, childcare, and school districts; public health; libraries; and local government, have been supporting and participating in the selection of local assessment tools as well as in the development of training and technical assistance to enable the valid and reliable use of assessment tools. These councils can apply for competitive grants to help in the purchase of approved assessment tools and support training. This coordinated community effort has multiple benefits, including supporting continuity of services and the transition of children between agencies in a local community. These efforts are assisted by a shared participation in the selection of a community-selected curriculum and assessment system based on a set of shared and common standards.

Although many states have elected to use a normreferenced tool to report child outcome data, the Kentucky system includes the use of an array of criterionreferenced curriculum-based tools. The use of curriculum tools in authentic settings with familiar materials and natural routines is aligned with recommended practice in assessment (NAEYC and NAECS/SDE, 2003; Neisworth and Bagnato, 2005) and produces functional outcome data that can be used to inform instruction, document child status, and generate accountability indices.

Curriculum-based tools include content and items that are aligned with the child's intervention or classroom curriculum. As Neisworth and Bagnato (2004) noted, "Linkages between assessment content and instructional content are essential to reveal program outcomes and impact" (p. 200). The potential for reliable and valid outcomes from these tools is greatly enhanced by the active involvement of families, teachers, and other team members. The use of measurement characteristics, such as graduated scoring options, also will help to provide more accurate reflections of child status. Furthermore, those tools with sufficient item density may also support test validity (especially at the lower limits or floor of the tool) for children with disabilities. In addition, unlike most norm- referenced tools, some of these instruments (i.e., AEPS; Hawaii Early Learning Profile [HELP]; Parks, 2006) were developed for children with disabilities and were field validated on children with disabilities.

Despite these advantages, a significant problem in the use of a criterion-referenced curriculum-based tool as an accountability index is the absence of a referent or norm necessary for comparison (Bagnato & Neisworth, 1983). These tools typically report the number of items or objectives a child achieves (not necessarily what should be achieved). One potential solution is the comparison of the achievement of children with disabilities with children without disabilities at two points in time, producing an index that can then be used as a measure of the efficiency of the curriculum and intervention (Bagnato & Neisworth, 1983). Other researchers (Kamphaus & Lozano, 1984) also have suggested that local norms may be an appropriate comparison group. Shinn (1989) stated,

Local norms are appealing because they are theoretically more representative of the milieu in which the child is currently functioning. . . . If nothing else, a child shares a common geographic region and community with the local reference groups that is shared at most by only a small portion of the national norming sampie. (p. 92)

These shared experiences are especially important for children from diverse cultural or minority populations. The collection of assessment data for all young children, not just children with disabilities or special needs, provides a ready-made comparison group for the development of program, district, and state norms. Data from each cohort and subsequent cohorts of children can be used to develop expectancy tables and to generate cut or criterion scores.

Good, Simmons, and Kame'enui (2001) suggested using what Thurlow and Thompson (1999) referred to as "assessment for accountability purposes" (p. 3), which might in fact be the component of the standards-based reform that has caused the most significant division of opinion across the country. As is often the case, the new federal mandate for the collection and reporting of child outcome data has preceded both the research and development of assessment tools for collecting and reporting child outcome data for accountability purposes. This leaves states in the uncomfortable position of developing accountability systems based on the best available resources and tools. Given these circumstances, Kentucky has chosen to develop a system that supports the use of high-quality, continuous assessment procedures and instruments that will provide teachers with ongoing data to improve instruction. At the same time, the state's responsibility is to continue to collect data to verify that this assessment data can reliably and validly be used on an aggregate level to document children's progress on both the Kentucky Standards and the OSEP outcomes and to make adjustments and changes as appropriate and/or as new resources become available.?

REFERENCES

Bagnato, S., & Neisworth, J. (1983). Monitoring developmental progress of young exceptional children: The curricular efficacy index. The Journal of Special Education, 17, 189-193.

Bodrova, E., Leong, D., & Shore, R. (2004). Child outcomes standards in Pre-K programs: What are standards; What is needed to make them work? National Institute for Early Education Research. Retrieved on October 21, 2006, from http://nieer.org/

Bricker, D., Capt, B., & Pretti-Frontczak, K. (2002). Assessment, evaluation, and programming system for infants and children (2nd ed.). Baltimore: Brookes.

Brigance, A. H. (2004). Brigance diagnostic inventory of early development-II. North Billerica, MA: Curriculum Associates.

Browder, D., Flowers, C., Ahigrim-Delzell, M. K., Spooner, F., & Algozzine, R. (2004). The alignment of alternative assessment content with academic and functional curricula. The Journal of Special Education, 37, 211-223.

Bronfenbrenner, U. (1993). The ecology of cognitive development: Research models and fugitive findings. In R. H. Wozniak & K. W. Fischer (Eds.), Development in context (pp. 3-44). Hillsdale, NJ: Erlbaum.

Burchinal, M. R., Roberts, J. E., Nabors, L. A., & Bryant, D. M. (1996). Quality of child care and infant cognitive and language development. Child Development, 67, 606-620.

Burns, M. S., Midgette, K., Leong, D., & Bodrova, E. (2003). Prekindergarten benchmarks for language and literacy: Progress made and challenges to be met. Retrieved on February 18, 2004, from http:/ /nieer.org/docs/indec.php?DocID=61.

Chazan-Cohen, R., Jerald, J., & Stark, D. R. (2001). A commitment to supporting the mental health of our youngest children. Zero to Three, 22, 4-12.

Clarke-Stewart, K. A., Gruber, C. P., & Fitzgerald, L. M. (1994). Children at home and in day care. Hillsdale, NJ: Erlbaum.

Cohen, M. A., & Gross, P. J. (1979). The developmental resource: Behavioral sequences for assessment and program planning. New York: Grune & Stratton.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Harcourt, Brace & Jovanovich. Deno, S. L. (2003). Developments in curriculum-based measurement. The Journal of Special Education, 37, 184-192.

Deno, S. L., Marston, D., & Tindal, G. (1986). Direct and frequent curriculum-based measurement: An alternative for educational decision making. Special Services in the Schools, 2, 5- 27.

Dichtelmiller, M. L., Jablon, J. R., Marsden, D. B., & Meisels, S. J. (2001). The work sampling system: Omnibus guidelines preschool through third grade. New York: Pearson Early Learning.

Dodge, D. T., Colker, L. J., & Heroman, C. (2001). The creative curriculum developmental continuum assessment system. Washington, DC: Teaching Strategies.

Dunn, L. (1993). Proximal and distal features of day care quality and children's development. Early Childhood Research Quarterly, 8, 167-192.

Feldman, A. B., Haley, S. M., & Coryell, J. (1990). Concurrent and construct validity of the Pediatric Evaluation of Disability Inventory. Physical Therapy, 70, 602-610.

Flowers, C., Browder, D., & Ahigrim-Delzell, L. (2006). An analysis of three states' alignment between language arts and mathematics standards and alternate assessments. Exceptional Children, 72, 201-215.

Good Start Grow Smart. (2002). Available from http:// www.whitehouse .gov/infocus/earlychildhood/toc.html

Good, R. H., Simmons, D. C., Se Kame'enui, E. J. (2001). The importance and decision-making utility of a continuum of fluency- based indicators of foundational reading skills for third-grade high- stakes outcomes. Scientific Studies of Reading, 5, 257-288.

Governor's Early Childhood Task Force. (1999). KIDS NOW: Kentucky invests in developing success. Frankfort, KY: Author.

Harbin, G., Rous, B., & McLean, M. (2005). Issues in designing state accountability systems. Journal of Early Intervention, 27, 137- 164.

High/Scope Educational Research Foundation. (2003). Preschool child observation record (2nd ed.). Ypsilanti, MI: Author.

Howes, C. (1997). Children's experiences in center-based child care as a function of teacher background and adult-child ratio. MerrillPalmer Quarterly, 43, 404-425.

Johnson-Martin, N. M., Attermeier, S. M., & Hacker, B. J. (2004). The Carolina curriculum for infants & toddlers with special needs (3rd ed.). Baltimore: Brookes.

Kamphaus, R. W., &c Lozano, R. (1984). Developing local norms for individually administered tests. School Psychology Review, 18, 213- 228.

Kendall, J. S. (2003, April). Setting standards in early childhood education. The first Years of School, 60, 64-68.

Kentucky Department of Education. (2005). State performance plan. Frankfort, KY: Author.

Kentucky Early Childhood Development Authority, Kentucky Revised Statute Chap. 211, [section] 120, 200.703 (2000).

Kentucky General Assembly. (1990). Kentucky education reform act of 1990. Gen. Assembly, Reg. Sess., Vol. 1.

Kontos, S., & Wilcox-Herzog, A. (1997). Research in review: Teachers' interactions with children: Why are they so important? Young Children, 52, 4-12.

Lewis, D. M., & Haug, C. A. (2005). Aligning policy and methodology to achieve consistent across-grade performance standards. Applied Measurement in Education, 18,11-34.

Lund, I., Rous, B. S., Purvis, H., & Moffett, C. L. (2003). Profiles on state pre-kindergarten standards and accountability systems: Technical report 2. Lexington: University of Kentucky, Interdisciplinary Human Development Institute, State Preschool Accountability Research Collaborative.

Macy, M. G., Bricker, D. D., & Squires, J. K. (2005). Validity and reliability of a curriculum-based assessment approach to determine eligibility for Part C services. Journal of Early Intervention, 28, 1-16.

McCormick, K., & Brennan, S. (2001). Mentoring the new professional in early childhood special education: The Kentucky Teacher Internship Program. Topics in Early Childhood Special Education, 21, 131-147.

McLaughlin, M. J., Henderson, K., & Rhim, L. M. (1998). Snapshots of reform: How five local districts are interpreting standards- based reform for students with disabilities. Alexandria, VA: National Association of State Boards of Education, Center for Policy Research on the Impact of General and Special Education Reform.

National Association for the Education of Young Children. (1997). NAEYC Position Statement: Developmentally appropriate practice in early childhood programs serving children birth through 8. Washington, DC: Author.

NAEYC and NAECS/SDE. (2002). Joint position statement: Early learning standards: Creating the conditions for success. Retrieved January 27, 2004, from http://www.naeyc.org/about/positions/pdf / position_statement.pdf

NAEYC and NAECS/SDE. (2003). Joint position statement: Early childhood curriculum, assessment, and program evaluation building an effective, accountable system in programs for children birth through age 8. Retrieved January 27, 2004, from http://www .naeyc.org/ resources/position_statements/CAPEexpand.pdf

Neisworth, J., & Bagnato, S. (2004). The mismeasure of young children: The authentic assessment alternative. Infants and Young Children, 17,198-212.

Neisworth, J., & Bagnato, S. (2005). DEC recommended practices: Assessment. In S. Sandall, M. L. Hemmeter, B. Smith, & M. McLean (Eds.), DEC recommended practices: A comprehensive guide. Longmont, CO: Sopris West.

Neuman, S. B., & Roskos, K. (2005). The state of state pre- kindergarten standards. Early Childhood Research Quarterly, 20, 125- 145.

Newberry, N., & Howard, M. (Eds.). (2005). Building a strong foundation for school success: Kentucky's early childhood quality selfstudy. Frankfort: Kentucky Department of Education.

Newborg, J. (2005). Battelle developmental inventory-2. Riverside, CA: Riverside Publishing.

No Child Left Behind Act of 2001, 20 U.S.C. 70 [section] 6301 et seq. (2002)

Parks, S. (2006). Inside HELP: Administration and reference manual for HELP. Palo Alto, CA: VORT Corp.

Peisner-Feinberg, E., Burchinal, M., Clifford, R., Culkin, M., Howes, C., Kagan, S., et al. (2001). The relation of preschool child- care quality to c

Source: Topics in Early Childhood Special Education

More News in this Category