July 1, 2008
Validation of the Systematic Screening for Behavior Disorders in Middle and Junior High School
By Caldarella, Paul Young, Ellie L; Richardson, Michael J; Young, Benjamin J; Young, K Richard
The Systematic Screening for Behavior Disorders (SSBD), a multistage screening system designed to identify elementary school- age children at risk for emotional and behavioral disorders, was evaluated for use with middle and junior high school students. During SSBD Stage 1, teachers identified 123 students in grades 6 through 9 with characteristics of internalizing and externalizing disorders. Teachers then completed SSBD Stage 2 behavior rating scales, the Teacher Report Form, and the Social Skill Rating System on 119 of these students identified as at-risk during Stage 1. Office discipline referrals and cumulative grade point averages for at-risk students were compared to those of students not designated by teachers. SSBD Stage 2 scores were compared with scores from the Teacher Report Form and Social Skill Rating System. Internal consistency and interrater reliability of the SSBD were also examined. Results provide evidence for the reliability and validity of SSBD ratings of early adolescent students. Keywords: at-risk populations, secondary education, emotional and behavioral disorders, assessment, adolescents, emotional and social competency, measurement
Outcomes for Students With EBD
Attempts to serve students with EBD in general education environments appear to often fall short, which may be one reason for these students' poor long-term outcomes. A recent article summarized some of the characteristic outcomes for students with EBD (Wagner et al., 2005); just about half (51%) drop out of school, which is the highest dropout rate of any educational disability category. Youth with EBD have lower grades and fail more classes. For secondary school students with EBD, 73% have a history of suspension or expulsion from school compared with only 22% of students in the general population. Youth with EBD are more likely than students in the general population to repeat a grade. Lane, Carter, Pierson, and Glaeser (2006) noted a higher rate of school absenteeism for students with EBD. Zigmond (2006) also found that many students with EBD are underemployed or unemployed after high school and that only a minority pursue postsecondary education.
As the Individuals With Disabilities Education Act (2004) criteria indicate, youth with EBD often have difficulty forming and maintaining relationships. Adolescent students with EBD have been found to have lower levels of social competence and higher levels of problem behavior than do students identified with learning disabilities (Lane et al., 2006). Cullinan and Sabornie (2004) have also noted that middle and high school students with EBD have been rated as having higher levels of social maladjustment and lower levels of overall competence than do students without EBD. Students with EBD appear to misinterpret social situations and have difficulty with interpersonal problem-solving skills (Lane, Wehby, & Barton-Arwood, 2005). Such difficulties can lead to rejection by peers (Murray & Greenberg, 2006) and increased association with other rejected peers (Kendziora, 2004), further leading to heightened risks for problematic outcomes. There is also increasing concern for students with internalizing behaviors, given the growing prevalence rates of these disorders and their link to disability and academic difficulties (Herman, Merrell, & Reinke, 2004).
Students with EBD are often viewed as less academically competent than students with learning disabilities (Lane et al., 2006) and may struggle due to their limited task completion, academic skill deficits, and lack of content knowledge (Lane et al., 2005). High school students with emotional disturbances have been found to display significant deficits in reading and math, similar to students with learning disabilities (Lane et al., 2006). The delayed academic skills of students with EBD tend to be stable in reading and written language, whereas deficits in mathematic skills tend to broaden over time, possibly due to the lack of enrollment in higher level math courses (Nelson, Benner, Lane, & Smith, 2004).
Services for Students With EBD
Providing educational services for students with EBD appears to be a complex and challenging process. General education teachers commonly report that students with EBD are among the least desirable to have in their classrooms (Soodak, Podell, & Lehman, 1998) and that they feel ill prepared to address the needs of such students (Cheney & Barringer, 1995; Cook, 2002). Students with EBD are more often placed in restrictive educational settings (e.g., residential schools, day treatment programs) than those of any other disability category (Gagnon & Leone, 2006).
Wagner and colleagues (2006) report on a variety of services for students classified as having EBD. Such students are more likely to receive instruction in general classrooms than in special education classrooms. Common academic accommodations provided to students with EBD include slower paced instruction, more time to take tests and complete assignments, and having tests read to them or modified in some way, despite limited evidence regarding the effectiveness of such accommodations with these students. Behavioral interventions or mental health services are implemented for only a minority of students with EBD, although they receive such services more frequently than do peers with other educational disabilities.
Additional findings regarding services for students with EBD have been noted by Wagner and colleagues (2005). Although the age of first diagnosis reported by parents is similar across disabilities, those with EBD receive special education services an average of more than 1 year later than those with other disabilities. Students classified as having EBD are less likely to receive preschool special education and early intervention services than are students with other disabilities, even when they may have been identified appropriately. Youth with EBD are also more likely than students in the general population to change schools due to district assignment rather than grade progress or family relocation. Parents with a student classified as EBD are more likely to work harder to secure services for their child, express dissatisfaction with their child's educational services, and pursue mediation or due-process hearings than are parents of students with other educational disabilities.
Given the lack of satisfaction with services for students with EBD, as well as their multiple negative and chronic educational outcomes, improved prevention and early identification efforts are needed. Once established, EBD appears to be quite difficult to treat; thus, there is a need to identify and serve such youth before their problems become entrenched. Effective prevention of EBD includes early identification and intervention with adolescent students who engage in concerning behavior patterns during their later school years (Forness, Kavale, MacMillan, Asamow, & Duncan, 1996). Screening can lead to such early identification, which, when combined with appropriate intervention, has been shown to address the educational challenges faced by such students (Kamps, Kravits, Stolze, & Swaggart, 1999). Students identified through the screening process as being at risk can become the target of proactive interventions, which may preclude the need for later referral for special education services.
Positive Behavior Supports
Screening is also an important component of the positive behavioral support (PBS) three-tiered model, which uses a preventive, teaching approach for creating positive school environments (Sugai, Horner, & Gresham, 2002). The model is based on research indicating that approximately 80% of students respond to universal interventions that explicitly teach and reinforce behavioral expectations. Targeted interventions can provide specific services and support to an estimated 10% to 15% of students who may have been labeled as at risk and projected to benefit from services such as small-group instruction in social skills. The more intensive individual level of support targets approximately 1% to 5% of students who require highly focused assessment and intervention services, including those with educational disabilities (Horner & Sugai, 2002; Sugai & Horner, 1999). Screening, identification, and treatment are vital to the success of a PBS program (Nelson, Benner, Reid, Epstein, & Currin, 2002). Effective screening and early identification of at-risk students may help to decrease the chances that EBD will develop and worsen over time. Successful prevention is more likely when screening efforts are in place to inform the selection of appropriate interventions. Effective treatment is dependent on accurate screening and identification, as screening can aid in determining which students will benefit from interventions at the targeted and individual levels of the PBS model (Nelson et al., 2002), thus serving a triage function. Providing students with individual interventions, when universal or small-group interventions would be sufficient, may not be a wise use of limited school resources. Therefore, screening becomes a vital beginning point in the process of meeting the needs of these students.
Screening for EBD
Universal screening differs from individual assessment because it includes a preliminary examination of the needs of all students in a school and has three characteristics (Glover & Albers, 2007). The first characteristic addresses the appropriateness of the screening instrument for the intended use; this includes being developmentally and contextually appropriate, complementing and supporting local service delivery models, and examining constructs relevant to a student's risk status. The second characteristic refers to technical adequacy, which focuses on the identification of a high proportion of students who need further assessment and intervention (i.e., true positives, or sensitivity) and a low proportion of students who do not need further services (i.e., true negatives, or specificity). The third characteristic addresses usability, including the cost of screening, administration time, data management requirements, and making efficient use of school resources. Usable screeners are also acceptable to stakeholders and sensitive to issues of diversity. Accurate and efficient screening also reveals whether the long-term outcomes for students identified and not identified are notably different (Hill, Lochman, Coie, & Greenberg, 2004).
Screening is intended to alleviate the "wait-to-fail" approach (Glover & Albers, 2007) often inherent in the special education process. With the most recent reauthorization of the Individuals With Disabilities Education Act, there is a targeted emphasis on prevention and early intervention through the response-to- intervention model (Fuchs, Mock, Morgan, & Young, 2003). This focus encourages the institutionalization of proactive, preventive efforts, rather than waiting for students to fail before addressing their needs (Albers, Glover, & Kratochwill, 2007). Such a proactive approach is found in the Systematic Screening for Behavior Disorders (SSBD; Walker & Severson, 1992), which was designed to aid in the identification of elementary school students at risk for EBD. The SSBD relies heavily on teacher nominations and ratings of students' behaviors based on the logic that classroom teachers have the training and experience to provide accurate information on their students' risk status. The SSBD uses a multiple gating system in which students have to pass through three gates (or stages) to warrant referral for school-based services.
Although there has been much written to encourage universal screening for early elementary school-age children with social and emotional concerns (Conroy et al., 2004; Walker, Cheney, Stage, & Blum, 2005), there is limited work on the development and validation of screening efforts for secondary school-age students (Lane, Wehby, Robertson, & Rogers, 2007). This is particularly unfortunate because students face a number of challenges as they transition from elementary to middle school, including larger schools; more impersonal, bureaucratic administrative procedures; and nonindividualized, departmentalized instruction (Eccles, Lord, & Midgely, 1991). These conditions may compound risk factors for those adolescents who are already socially or emotionally vulnerable. Thus, adequate screening of early adolescents is warranted.
Previous research (Cheney & Barringer, 1995) using the SSBD in middle schools suggested that this instrument might be useful in identifying early adolescent students who are at risk. Other recent work using the SSBD with a high school population, although promising, has been limited by the lack of specific reliability and validity studies of the instrument in secondary school settings (Lane et al., 2007). This study was conducted to examine whether it would be effective to use the SSBD to identify middle and junior high school students who might be at risk for EBD. The specific research questions addressed in this study were: (a) Is there evidence for the concurrent validity of the SSBD Stage 1 nominations when used with middle and junior high school students? (b) Is there evidence for the reliability of the SSBD Stage 2 scores when used with middle and junior high school students? (c) Is there evidence for the convergent and discriminant validity of SSBD Stage 2 scores when used with middle and junior high school students?
A middle school and a junior high school located in suburban and rural areas of Utah were the settings for this study. During the year of the study, 1,072 students were enrolled in the middle school, which included Grades 6 and 7. In this school, 52% of the students were male, 90% were Caucasian, 9% were Hispanic, and less than 1% were of other ethnicities. Approximately 37% of the students qualified for free or reduced lunch. Of 52 teachers at the middle school, 84% participated in the study.
During the year of the study, 1,074 students were enrolled in the junior high school, which included Grades 8 and 9. In this school, 53% of the students were male, 90% were Caucasian, 6% were Hispanic, and 1% each were African American, Asian, Native American, or Pacific Islander. Approximately 26% of these students qualified for free or reduced lunch. Of 50 teachers at the junior high school, 84% participated in the study.
Both of these schools were in the initial stages of implementing a schoolwide PBS model. This model included working with faculty and staff to include social skills instruction in all classrooms, develop and teach schoolwide goals and expectations, and increase the level of positive feedback students were receiving at school. Use of the SSBD was part of the identification process for students who might benefit from additional targeted interventions in a small- group format. Teachers were asked to consider all of their students when making nominations during SSBD Stage 1.
The primary participants in this study were 123 student (59 at the middle school, 64 at the junior high school) who had been nominated and then ranked by their teachers using the SSBD and for whom parental consent for participation was obtained. During SSBD Stage 1, teachers were asked to nominate up to 5 students in both internalizing and externalizing categories, rather than 10 students as suggested by the SSBD manual. On average, student participants were nominated by two teachers. Teacher rankings were then scored in reverse order starting with 5 being the most concerning student down to 1 being the least concerning. Once the teacher rankings were scored, they were combined such that a student who was nominated and then ranked as the most concerning (5) by two teachers would have a combined score of 10. The use of combined teacher rankings is based on the assumption that students with multiple nominations and high rankings might indicate more extensive behavior problems across settings or classrooms in these schools. Because some students with behavioral and emotional concerns might not come to the attention of one teacher (especially students with internalizing behaviors), they are less likely to escape the notice of multiple teachers. In alignment with the SSBD elementary school procedures, students who were not ranked as one of the top-three most concerning students in either the internalizing or externalizing category by at least one of their teachers were excluded from participation in this study. Students who qualified for this study had an average combined score of 7.42 (SD = 3.96) in this nomination stage.
At the middle school, 76% of the participating students were male, 86% were Caucasian, 12% were Hispanic, and 2% were Native American. Of the students enrolled at this school, 232 (22%) were initially nominated by teachers using the SSBD Stage 1. Parents of 127 of the students who were nominated by multiple teachers, or ranked as one of the top-three externalizing or internalizing students by their teachers, were contacted for possible participation. Consent for participation was obtained for 60 (56%) of these students.
At the junior high school, 80% of the participating students were male, 95% were Caucasian, and 5% were Hispanic. Of the students enrolled at this school, 16% were initially nominated by teachers using the SSBD Stage 1. Parents of 129 of the students who were nominated by multiple teachers or were ranked as one of the top- three externalizing or internalizing students by their teachers were contacted for possible participation. Consent for participation was obtained for 66 (51%) of these students.
All student participants met SSBD criteria for moving on to Stage 2. The second stage of the SSBD has normed cutoff scores (Critical Events Index and Combined Frequency Index) for first-through sixth- grade students to estimate level of risk and continuing on to Stage 3. Approximately 55% of the student participants scored above these cutoffs and would have entered Stage 3 if it had been included as part of this study. However, because these cutoff scores were established using data from elementary school students, they may not be accurate for middle and junior high school students. Measures
Three formal behavior rating scales were used in this study: the SSBD (Walker & Severson, 1992), the Teacher Report Form (TRF; Achenbach & Rescorla, 2001), and the Social Skills Rating System (SSRS; Gresham & Elliot, 1990). Classroom teachers completed all scales. Two additional measures were obtained from existing school data sources (a district database): office disciplinary referrals (ODR) and cumulative grade point average (GPA).
SSBD Stage 1. The SSBD is multistaged process for identifying elementary students who are at risk for behavior disorders. Stage 1 asks classroom teachers to nominate and rank order students from their classes who have internalizing or externalizing behaviors. Definitions of internalizing and externalizing behavior problems, along with examples and nonexamples, are given to teachers. (Internalizing includes "being shy, timid, and/or nonassertive,""preferring to play or spend time alone," or "acting in a fearful manner^'; externalizing is manifested by "displaying aggression toward objects or persons,""not complying with teacher instructions or directives," or "defying the teacher.") In the standard administration of the SSBD in elementary school settings, teachers consider all students in their classrooms and then nominate 10 students in each category and rank them according to the extent to which they exhibit the provided descriptive characteristics for internalizing and externalizing behaviors in relation to other students. The three top-ranked students in each category are then included in the Stage 2 screening process.
SSBD Stage 2. The SSBD Stage 2 includes the Critical Events Index (CEI) and Combined Frequency Index (CFI) for Adaptive and Maladaptive Behavior. The latter includes two separate subscales: Adaptive Student Behavior and Maladaptive Student Behavior. The CEI consists of a h'st of both internalizing and externalizing behaviors. (Internalizing includes "exhibits painful shyness,""is teased, neglected and/or avoided by peers," and "has severely restricted activity levels"; externalizing contains "steals,""physically assaults an adult," and "damages others' property.") These items are presented in a checklist format on which the teacher indicates the presence or absence of the behavior. The CFI for Adaptive and Maladaptive Behavior lists behaviors that are rated on a 5-point Likert-type scale ranging from never to frequently. The Adaptive Student Behavior subscale includes items such as "follows established classroom rules" and "initiates positive social interactions with peers." The Maladaptive Student Behavior subscale includes such behaviors as "refuses to participate in games and activities with other children at recess" and "uses coercive tactics to force the submission of peers; manipulates, threatens, etc." The SSBD Stage 2 also has normed CEI and CFI cutoffs for estimating level of risk and movement on to Stage 3.
SSBD Stage 3. SSBD Stage 3 consists of obtaining independent classroom observations of behavior for students who meet or exceed normative criteria on the SSBD Stage 2. These include multiple observations of Academic Engaged Time and Peer Social Behavior. Students at this stage are observed both in the classroom and on the playground by an independent observer (other than the nominating teacher). These independent observers are usually other school professionals such as school psychologists, counselors, or resource teachers who have been trained in the SSBD Stage 3 observation system.
The SSBD has been standardized and normed; reviewers have suggested that it has evidence of reliability and validity for identifying elementary students at risk for EBD (Kelley, 1998; Zlomke & Spies, 1998). A series of studies completed in conjunction with development and validation of the instrument presented in the SSBD manual yielded reliability estimates for its use at the elementary level. Internal consistency (a) was estimated above .80 for the Stage 2 subscales Adaptive and Maladaptive Student Behavior. Elementary test-retest reliability (p) for Stage 1 reported rankings of internalizing behavior as .72 and externalizing behavior as .79. During the instrument development phase, interrater agreement (Spearman p) on the internalizing and externalizing dimensions of Stage 1 ranged from .82 to .94 (see Walker & Severson, 1992).
Teacher Report Form. The TRF (Achenbach & Rescorla, 2001) includes teacher ratings of 118 problem behaviors. The items are grouped into several subscales: Withdrawn, Somatic Complaints, Anxious/Depressed, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior, and Aggressive Behavior. These subscales are then grouped into Internalizing (Withdrawn, Somatic Complaints, and Anxious/Depressed) and Externalizing (Rule-Breaking Behavior and Aggressive Behavior) scales. The Internalizing and Externalizing scales were of primary interest for this study. As reported in the TRF test manual, internal consistency (a) estimates have been reported above .80 for most of the subscales and above .90 for several. An exception is the Somatic Complaints subscale (alpha = .72). Pearson's r, which was also used to estimate cross- informant agreement on the subscales, ranged from .28 (Somatic Complaints) to .69 (on several subscales). Most subscales were in the upper .50 and .60 range for crossinformant agreement. These estimates were obtained from a normative sample of 2,319 children, 48% male and 52% female, distributed across the Northeastern, Midwestern, Southern, and Western regions of the United States, with a somewhat higher representation from the South (36%). Middle socioeconomic status had the largest representation (46%), followed by upper class (38%). The majority of students were non-Latino White (72%), followed by African American (14%), Latino (7%), and mixed or other (7%; Achenbach & Rescorla, 2001).
Social Skills Rating Scale. The SSRS (Gresham & Elliot, 1990) is a 50-item Likert-type rating scale. Two scale scores are included in the measure: Social Skills and Problem Behaviors. The measure also includes importance rankings on the Social Skills scale and the Academic Competence scale, used for ranking students relative to other class members. The Problem Behaviors scale, which includes Internalizing and Externalizing subscales, was of primary interest for this study. Internal consistency (a) of secondary teacher ratings for the Problem Behaviors scale scores have been reported as .89 for the Externalizing subscale score and .80 for the Internalizing subscale score. For these analyses, 51 secondary teachers (314 student ratings) were drawn from an overall standardization sample that included 259 preschool, elementary, and secondary teachers. Demographic data reported on the overall teacher sample indicated that the sample was 88% female, 90% White, 40% from Southern states, and 36% from North Central states. However, the majority (80%) of these teachers were working at elementary schools and thus were not included in reliability analyses for the secondary level (Gresham & Elliot, 1990).
Office disciplinary referrals and grade point average. Disorderly conduct ODR, attendance ODR, and cumulative GPA for each of the students participating in the study, as well as schoolwide averages on these measures, were examined. Disorderly conduct ODR represented office referrals related to behavioral conduct while at school. Attendance ODR represented office referrals for attendance problems such as unexcused absences and truancy. Both schools used a district- maintained network database for recording and storing student information. ODR and GPA information was obtained from this database. The ODR reporting system included specific categories of behaviors that could be separated into attendance-related referrals and conduct-related referrals.
Teacher rankings of students on the SSBD Stage 1 were obtained during faculty meetings. Teachers were instructed on how to complete the forms, and forms were collected at the end of the meetings. Because this study examined the use of the SSBD for middle and junior high school students, modifications were made to the Stage 1 procedure. Adolescents typically meet with several teachers during the day, in contrast to elementary students who have one primary teacher. Teachers were asked to consider all the students they teach during the day. However, because of these differences, several teachers may have nominated the same student; in such cases, teacher rankings were combined. School district administrators required that parental consent be obtained before asking teachers to complete the SSBD Stage 2, due to concerns regarding the more intrusive nature of the Stage 2 items.
SSBD Stage 2 forms along with TRF and SSRS forms were prepared in a packet and delivered to teachers of students for whom parental consent to participate was obtained. Teachers were given detailed instructions on completing the measures when they were delivered. With permission, schoolwide ODR (averages per student for attendance referrals and disorderly conduct referrals) and schoolwide GPA were used from the school district database. Individual ODR and GPA data also were obtained for students for whom consent was given. SSBD Stage 3 observations were not conducted on the identified students because this study focused on evaluating only Stages 1 and 2. However, Stage 3 may provide additional valuable information (e.g., multimethod, nonclassroom settings) at the middle and junior high school level.
To address the research questions, several analyses were conducted. These included two analyses of SSBD Stage 1, a comparison of selected students with the general school population, and a comparison of students identified as internalizing with students identified as externalizing. Stage 2 analyses addressed reliability and examined correlations with other screening measures (TRF and SSRS). Each of these analyses is described in greater detail below. Analyses of SSBS Stage 1. To determine if students nominated by teachers as externalizing or internalizing represented a different group than nonnominated students, t tests were conducted to compare nominated students at Stage 1 with schoolwide averages on ODR (for attendance and disorderly conduct) and GPA. Nominated students were also examined to determine the percentage of these students (both internalizing and externalizing) who reached or exceeded 1.5 standard deviations from the schoolwide mean. The 1.5 standard deviation cutoff was chosen because it corresponds to the Achenbach "borderline" categorization (Achenbach & Rescorla, 2001). This analysis allowed for an indication of what percentage of the nominated students likely accounted for significant f-test differences between nominated students and schoolwide means.
To further assess the validity of the Stage 1 externalizing and internalizing descriptions, students nominated as having internalizing behaviors were compared, using t tests, on SSBD Stage 2, TRF, SSRS, ODR, and GPA to students nominated as having externalizing problems. These analyses were conducted to examine whether teacher nominations at SSBD Stage 1 corresponded, on average, with other measures of student behavior. To explore the validity of the nomination procedure, students nominated in the externalizing category should also be rated higher on other externalizing scales, and students nominated as internalizing should be rated higher on other internalizing scales. Comparisons of students nominated as externalizing or internalizing on scales not specific to these behavior categorizations were considered exploratory (e.g., GPA, attendance, ODR, etc.).
Analyses of SSBS Stage 2. Next, internal consistency and interrater (teacher) reliability of SSBD Stage 2 were examined. Cronbach's alpha was calculated to examine internal consistency on each Stage 2 scale. In addition to calculating the alpha for the existing Maladaptive and Adaptive scales, the SSBD Critical Events scale was separated into Internalizing and Externalizing subscales. This was done because the Critical Events scale contains a mixture of externalizing and internalizing items that should not necessarily be consistent with each other. Items on the SSBD Critical Events scale were selected as either internalizing or externalizing by performing f-test comparisons of each item between students nominated as internalizing or externalizing on the SSBD Stage 1. Only items that significantly discriminated between these two teacher-nominated groups were included in the Critical Events Internalizing and Externalizing subscales. It should be noted that these Critical Events subscales are not included in the SSBD manual and were created for the purpose of more fully evaluating the use of the tool with middle and junior high students.
Intraclass correlations (ICCs) were also calculated to examine interrater reliability. Interrater reliability estimates in this study differ from typical estimates because they represent a correlation between teacher ratings obtained at different times and in different classroom settings. Typically, interrater reliability is obtained when two observers rate behavior at consistent times and in consistent settings. This traditional type of interrater reliability estimate is not attainable with a typical teacher- rating procedure.
Convergent validity and discriminant validity of the SSBD Stage 2 were also examined by obtaining Pearson's r estimates of the SSBD scales in relation to the TRF, SSRS, ODR, and GPA. These analyses allowed for comparisons of correlation coefficients between the Externalizing and Internalizing scales on the SSBD and on similar scales from the other measures to determine whether correlation coefficients were significant and in the expected direction (e.g., positive and significant correlations between similar scales and negative or nonsignificant correlations between dissimilar scales). In addition, these analyses allowed us to explore if the Adaptive and Maladaptive scales of the SSBD, GPA, and ODR were related to the Internalizing and Externalizing scales of the SSBD Stage 2.
Nominated Students Versus Schoolwide Averages
We conducted one-sample t tests using schoolwide means as test values to examine the concurrent validity of the SSBD Stage 1 as a screening tool in middle and junior high schools. Students nominated at SSBD Stage 1 were compared to schoolwide averages on number of disorderly conduct ODR and attendance ODR and on cumulative GPA. Analyses revealed statistically significant (p
Individual ODR and GPA data for nominated students were also examined to determine what percentage of students exceeded 1.5 standard deviations from the population mean on at least one of these measures (GPA, attendance ODR, and disorderly conduct ODR). As noted, the 1.5 standard deviation cutoff was chosen because it corresponds to the Achenbach "borderline" categorization (Achenbach & Rescorla, 2001). A total of 55% of nominated students exceeded this cutoff on at least one of the measures. When examined separately, 42% of students nominated as internalizing and 65% of students nominated as externalizing exceeded this cutoff on at least one measure.
In addition, the percentage of students who exceeded this 1.5 standard deviation cutoff was calculated for internalizing and externalizing students who either met or did not meet criteria for proceeding on to Stage 3. A total of 48% of internalizing students and 73% of externalizing students who met the criteria for moving on to Stage 3 exceeded this cutoff on GPA, attendance ODR, or disorderly conduct ODR. Twenty-nine percent of internalizing students and 63% of externalizing students who did not meet the Stage 3 criteria also exceeded these cutoffs.
Externalizing Versus Internalizing Students
Students with internalizing and externalizing behaviors identified through Stage 1 of the SSBD were compared, using independent t tests, on the following measures: SSRS and TRF internalizing and externalizing scores, SSBD Stage 2 Adaptive and Maladaptive scores, total ODR, and cumulative GPA. With the exception of GPA, these measures revealed statistically significant differences between students identified by SSBD Stage 1 as internalizing and externalizing (see Table 2). A negative t value indicates that externalizing students were rated higher on a given scale, whereas a positive t value indicates that internalizing students were rated higher.
Internal Consistency and Interrater Reliability
Internal consistency of the SSBD Stage 2 was evaluated for each of the scales included in this stage of the measure using Cronbach's alpha, which is comparable to a correlation coefficient measuring the strength of the relationship between items on a scale. Using average ICC, interrater reliability of the SSBD Stage 2 scales was calculated for 82 students who were each rated by two of their teachers (see Table 3). ICC is also comparable to a correlation coefficient measuring the strength of the relationship between ratings from the teachers.
Convergent and Discriminant Validity of SSBD Stage 2
Finally, we evaluated convergent and discriminant validity of the SSBD Stage 2 scales by obtaining Pearson's correlations between each of these scales and the relevant TRF (internalizing and externalizing), SSRS (internalizing and externalizing), and school- based (GPA and ODR) measures. Most correlations were statistically significant beyond the .01 or .05 level. Evidence of convergent and discriminant validity is indicated by positive and negative values, respectively (see Table 4). For example, SSBD Stage 2 scales associated with internalizing behaviors correlated positively with internalizing scales on the other measures and correlated negatively with externalizing scales on the other measures.
This study examined evidence of reliability and validity for the use of the SSBD in middle and junior high schools. The specific research questions addressed (a) the concurrent validity of the SSBD Stage 1 nominations, (b) the reliability of the SSBD Stage 2 scores, and (c) the convergent and discriminant validity of SSBD Stage 2 scores. Beyond the SSBD manual and a study by McKinney, Montague, and Howcutt (1998), both of which were completed in elementary school settings, there appear to be no published articles that have evaluated the psychometric properties of the SSBD in the comprehensive manner used in the present study.
This study provided preliminary empirical support for using the SSBD as an emotional and behavioral screening tool for early adolescent students. First, evidence supporting the concurrent validity of the SSBD Stage 1 teacher nomination procedure with middle and junior high school students was provided. Students identified as being at risk for EBD were found to differ significantly from students who were not identified. The at-risk students had a significantly higher number of ODR and significantly lower GPAs than did students who were not in the at-risk category. These results are consistent with other results in the literature (Lane et al., 2005; Lane et al., 2006; Nelson et al., 2004; Wagner et al., 2005), suggesting that youth at risk for EBD tend to display higher levels of problem behaviors and lower levels of academic achievement than those who are not at risk. A closer examination revealed that some nominated students did not have significantly elevated numbers of ODR or significantly lower GPAs (using a 1.5 standard deviation cutoff). Behaviors that teachers were seeing in these students possibly were not problematic enough to be reflected in increased ODR or reduced GPA. This was likely particularly true of the students identified as internalizing, as such behaviors are not necessarily incompatible with good grades and nondisruptive conduct. Indeed, these students may suffer more from isolation than from academic or interpersonal conflict problems. In addition, it appears that not all behaviors considered by teachers to be externalizing necessarily reach levels high enough to interfere with academic success or to result in removal from the classroom. Many of these behaviors may manifest as low-level annoyances to teachers and fellow students and could be interpreted as false positives but may escalate if not addressed. When the consequences for underidentification are great, it may make sense to sacrifice precision for inclusion during screening (Glover & Albers, 2007).
Another possibility is that some students exceeded these cutoffs but may not have been nominated by teachers. In this study, individually identifiable ODR and GPA data could not be analyzed without parental consent for students not participating in the study. This is an empirical question that could be addressed in future research; however, obtaining parental consent to access these students' data was difficult in the settings where this study was conducted.
Further evidence of the concurrent validity of the SSBD Stage 1 procedure was found in the results of comparisons with other well- established measures of emotional and behavioral risk. Specifically, teachers' initial nominations of internalizing and externalizing students corresponded to ratings on the Internalizing and Externalizing subscales of both the TRF and SSRS. Differences between the groups were in the expected direction: that is, at-risk internalizing students were higher on the Internalizing subscales of the TRF and SSRS, whereas at-risk externalizing students were higher on the Externalizing subscales of the TRF and SSRS, as well as on ODR.
The results of this study also provide support for the reliability of the SSBD Stage 2 scores with middle and junior high school students. What constitutes sufficient reliability depends on the importance of the decision to be made, the likelihood of future assessments, and the existence of other available data (e.g., ODR and GPA) that could help in making decisions regarding students' risk status. When an important decision is to be made, such as whether a student is at high risk for EBD, researchers and practitioners should make use of all available data to help identify such students. The SSBD appears to offer sufficient reliability evidence to contribute to screening efforts.
Support for the internal consistency of the Stage 2 instrument was evident in the relatively high alpha coefficients for both the Maladaptive and Adaptive subscales, yielding results similar to those found for use of the SSBD in elementary settings (Walker & Severson, 1992). The newly created Critical Events Externalizing subscale results also showed promise with a respectable (.75) alpha coefficient. The Critical Events Internalizing subscale results fell short, with a lower (.57) alpha coefficient, possibly due to the small number of items comprising this subscale and to the exclusive focus on student internalizing symptoms that are often more difficult to measure using parent or teacher rating scales (Merrell, 2003).
Study results also supported the interrater reliability of the SSBD Stage 2. Analyses of the SSBD Maladaptive and Critical Events Internalizing subscales scores both yielded relatively high ICCs given the different times and settings in which teachers interacted with these students. These findings indicated that teachers had reasonably high agreement in identifying at-risk students. Analysis of SSBD Critical Events Externalizing subscale scores yielded a more moderate correlation, whereas the Adaptive subscale showed a lower correlation. The interrater reliability results are generally lower than those found for use of the SSBD in elementary settings (Walker & Severson, 1992), possibly due to the fact that in middle and junior high school settings students are typically with five to seven teachers during the school day. All correlations were statistically significant, despite the fact that teachers observed students in different contexts and for relatively brief periods of time compared to in elementary settings.
Finally, the results of this study provide support for the convergent and discriminant validity of the SSBD Stage 2 in middle and junior high school settings. Convergent validity evidence examines whether instruments are measuring the same construct, whereas discriminant validity examines whether instruments are measuring different constructs. Convergent validity was suggested by the significant positive correlations between the SSBD Critical Events Internalizing subscale score and both the TRF and SSRS Internalizing subscale scores. Convergent validity was similarly suggested by the significant positive correlations between the SSBD Critical Events Externalizing subscale score and both the TRF and SSRS Externalizing subscales scores and the total number of ODR. The SSBD Adaptive subscale scores also correlated positively with GPA, whereas the Maladaptive scores correlated positively with number of ODR.
Evidence for discriminant validity was supported by the significant negative correlations between the SSBD Critical Events Internalizing subscale score and both the TRF and SSRS Externalizing subscale scores and number of ODR. The significant negative correlations between the SSBD Critical Events Externalizing subscale scores and both the TRF and SSRS Internalizing subscale scores provided additional evidence of discriminant validity. The SSBD Adaptive subscale scores also correlated negatively with number of ODR, whereas the SSBD Maladaptive subscale scores correlated negatively with GPA. These negative correlations suggest that the internalizing and externalizing scales represent not only different constructs but, in some cases at least, divergent constructs. The results of the present study, while using slightly different concurrent measures, were generally similar to those found by others examining the validity of the SSBD in elementary settings (McKinney et al., 1998; Walker & Severson, 1992).
Implications for Practice
This study appears to be the first to attempt a specific evaluation of the psychometric properties of the SSBD when used to screen a middle and junior high school population. In this regard, it offers an important initial examination of the potential for using the instrument to identify early adolescent youth who may be at risk for EBD. The SSBD appears to offer an efficient and valid manner of screening all students and thus has important implications for school staff and researchers.
Stage 1 of the SSBD provides an initial gate for middle and junior high school teachers to identify any of their students who may be at risk for EBD; thus, each student could potentially be nominated by all of his or her teachers, providing a wide array of observations and perceptions of student behavior. Other screening measures might require teachers to provide ratings for all students they teach, which is unrealistic if a teacher has multiple classes each with 20 to 30 students over the course of a day.
Another important implication of this study is that, based on comparisons with school data (ODR, attendance, GPA), teachers were able to identify a significant number of students who had verifiable problems. Some of the identified students did not exceed school norms; however, this may be due to the fact that teachers were sensitive to problems not reflected in the school data (e.g., internalizing behaviors). This is consistent with a preventive approach to EBD. The use of the SSBD allows for the inclusion of students who would not be identified using school data alone. This finding also supports the work of Martin, Hooper, and Snow (1986), which noted that teachers may be considered expert informants regarding child and adolescent behavior.
The results of this study suggest that the SSBD appears to possess sufficient evidence of reliability and validity to justify its experimental use with middle and junior high school students. For example, the SSBD may be used to identify appropriate participants for targeted or individual PBS interventions focusing on adolescent students at risk for EBD. Compared to other measures (e.g., the TRF or SSRS), the instrument offers a less-resource- intensive screening option for middle and junior high school staff. Combining the results of SSBD screening with other student data, such as grades, ODR, and attendance, would strengthen screening efforts with adolescents. Screening may also begin to address the extremely low number of students served as EBD by increasing the identification of and treatment for the estimated 20% of students with mental health needs who are currently underserved (see Mash & Dozois, 2002; Walker, Nishioka, Zeller, Severson, & Feu, 2000). Knowing the impact that unidentified emotional and behavioral problems can have on student outcomes, screening becomes crucial.
Implications for Future Research
This study did not fully investigate the reliability of the SSBD with middle and junior high school students. Although the SSBD Stage 1 does not lend itself to internal consistency evaluation, due to the nature of the ranking system, test-retest reliability of the scale can be investigated, as demonstrated by Walker and Severson (1992) in their SSBD validation studies with elementary school students. Future research on the test-retest reliability of the SSBD Stage 1 (and Stage 2) with adolescent students could help further establish the appropriate use of the tool with this population. Walker and Severson (1992) stated that internal consistency reliability coefficients were not calculated on the SSBD CEI because of the divergent nature of the scale (e.g., assessing both internalizing and externalizing domains). Although the findings of the present study should be interpreted cautiously, they nevertheless extend the psychometric evaluation of the SSBD by empirically separating out the CEI internalizing and externalizing items and evaluating these two newly created subscales. Additional focus on the reliability and validity of the CEI subscales attempting to replicate the findings of the current study could help extend the psychometric evaluation of the SSBD.
This study used a modification of the SSBD procedure whereby rankings from multiple teachers on Stage 1 were combined. This adaptation seemed reasonable given the nature of middle and junior high school in which students have five to seven teachers. More research is needed to further explore the implications of this modification and other ways to adapt the SSBD to identify at-risk adolescents.
The percentage of students nominated in SSBD Stage 1 at the two schools averaged 19%, which is consistent with prevalence estimates of mental health found in the literature (Mash & Dozois, 2002; Walker et al., 2000). Of the students in the present study who were screened at Stage 2, approximately 51% of externalizing and 60% of internalizing students would have reached Stage 3 using the elementary school CEI and CFI normative cutoff criteria. In comparison, approximately 34% of externalizing and 29% of internalizing students reached Stage 3 in studies reported in the SSBD manual (Walker & Severson, 1992). These results suggest that the procedure used in this study may have been more inclusive at Stage 2, likely resulting in fewer false negatives and more false positives. However, the percentage of students who would have reached Stage 3 is difficult to interpret because of the different student population being studied. More research establishing norms using the SSBD with adolescent students would help to clarify this issue.
As noted earlier, an effective universal screening instrument has three characteristics: appropriateness, technical adequacy, and usability (Glover & Albers, 2007). This study focused primarily on aspects of the appropriateness and technical adequacy of the SSBD in middle and junior high school settings. Further research on these characteristics, especially on the usability of the SSBD in such settings, is warranted.
A final implication for future research is the gathering and analysis of longitudinal data on the identified students. If the SSBD accurately identifies at-risk students, these students would be expected to have poorer long-term outcomes than would students not identified. We are currently tracking screened students over multiple years, thus investigating the predictive validity of the SSBD with this population. Determining which SSBD items (and other variables) best predict those positively screened youth who go on to develop more severe emotional and behavioral issues will be helpful in understanding the utility of the SSBD.
A major limitation of this study was the sample of students who were rated. First, the sample size was relatively small for making conclusive investigations of the validity of the SSBD with middle and junior high school students. The fact that all students lived in one school district in the intermountain West also meant a lack of diversity of the sample. Additional studies with a broader scope, examining the use of the SSBD with larger groups of adolescent students with more diverse demographic variables, would be helpful.
Similarly, there were a relatively low number of minority students in this study; future research could benefit from inclusion of a larger percentage of minority students to determine whether the SSBD provides evidence of reliability and validity for use with more ethnically diverse students. Because no high school students were included in this study, future research examining the performance of the SSBD with an older population of students would also be an important contribution to the literature.
The results of this study, although encouraging, should be considered preliminary. Future research on the SSBD should consider using a more national sample and making procedural modifications to reflect middle and junior high school settings (e.g., students having multiple teachers, teachers having more than 100 students). Completing an item analysis to ensure that developmentally appropriate items are included would also help to fully endorse the tool as a reliable and valid screener in these settings.
Achenbach, T. M., & Rescorla, L. A. (2001). Manual for the ASEBA school-age forms & profiles. Burlington: University of Vermont, Research Center for Children, Youth, & Families.
Albers, C. A., Glover, T. A., & Kratochwill, T. R. (2007). Introduction to the special issue: How can universal screening enhance educational and mental health outcomes? Journal of School Psychology, 45, 113-116.
Cheney, D. A., & Barringer, C. (1995). Teacher competence, student diversity, and staff training for the inclusion of middle school students with emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 3, 174-182.
Conroy, M. A., Hendrickson, J. M., & Hester, P. P. (2004). Early identification and prevention of emotional and behavioral disorders. In R. B. Rutherford, M. M. Quinn, & S. R. Mathur (Eds.), Handbook of research in emotional and behavioral disorders (pp. 199-215). New York: Guilford.
Cook, B. G. (2002). Inclusive attitudes, strengths, and weaknesses of pre-service general educators enrolled in a curriculum infusion teacher preparation program. Teacher Education and Special Education, 25, 262-277.
Cullinan, C., & Sabornie, E. J. (2004). Characteristics of emotional disturbance in middle and high school students. Journal of Emotional and Behavioral Disorders, 12, 157-167.
Eccles, J. S., Lord, S., & Midgely, C. (1991). What are we doing to early adolescents? The impact of educational contexts on early adolescents. American Journal of Education, 99, 521-542.
Forness, S. R., Kavale, K. A., MacMillan, D. L., Asamow, J. R., & Duncan, B. B. (1996). Early detection and prevention of emotional or behavioral disorders: Developmental aspects of systems of care. Behavioral Disorders, 21, 226-240.
Fuchs, D., Mock, D., Morgan, P. L., & Young, C. L. (2003). Responsiveness-to-intervention: Definitions, evidence, and implications for the learning disabilities construct. Learning Disabilities Research and Practice, 18, 157-171.
Gagnon, J. C., & Leone, P. E. (2006). Elementary day and residential schools for children with emotional and behavioral disorders: Characteristics of educators and students. Education and Treatment of Children, 29(1), 51-78.
Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117-135.
Gresham, F. M., & Elliot, S. N. (1990). Social Skills Rating System manual. Circle Pines, MN: American Guidance Service.
Herman, K. C., Merrell, K. W., & Reinke, W. M. (2004). The role of school psychology in preventing depression. Psychology in the Schools, 41, 763-775.
Hill, L. G., Lochman, J. E., Coie, J. D., & Greenberg, M. T. (2004). Effectiveness of early screening for externalizing problems: Issues of screening accuracy and utility. Journal of Consulting and Clinical Psychology, 72, 809-820.
Horner, R. H., & Sugai, G. (2002). School-wide positive behavior support: Implementer blueprint and self-assessment. Eugene, OR: OSEP Center on Positive Behavior Support.
Individuals With Disabilities Education Improvement Act of 2004,20 U.S.C. [section] 1400 et seq. (2004) (reauthorization of the Individuals With Disabilities Education Act of 1990).
Kamps, D., Kravits, T., Stolze, J., & Swaggart, B. (1999). Prevention strategies for at-risk students and students with EBD in urban elementary schools. Journal of Emotional and Behavioral Disorders, 7, 178-188.
Kauffman, J. (1999). How we prevent the prevention of emotional and behavioral disorders. Exceptional Children, 65, 448-468.
Kelley, M. L. (1998). Review of the Systematic Screening for Behavior Disorders. In J. C. Impara, B. S. Plake, & L. L. Murphy (Eds.), The thirteenth mental measurements yearbook (pp. 994-995). Lincoln, NE: Buros Institute.
Kendziora, K. T. (2004). Early intervention for emotional and behavioral disorders. In R. B. Rutherford, M. M. Quinn, & S. R. Mathur (Eds.), Handbook of research in emotional and behavioral disorders (pp. 32-53). New York: Guilford.
Lane, K. L., Carter, E. W., Pierson, M. R., & Glaeser, B. C. (2006). Academic, social, and behavioral characteristics of high school students with emotional disturbances or learning disabilities. Journal of Emotional and Behavioral Disorders, 14, 108- 117.
Lane, K. L., Wehby, J., & Barton-Arwood, S. M. (2005). Students with and at risk for emotional and behavioral disorders: Meeting their social and academic needs. Preventing Social Failure, 49(2), 6- 9.
Lane, K. L., Wehby, J., Robertson, E. J., & Rogers (2007). How do different types of high school students respond to schoolwide positive behavior support program? Journal of Emotional and Behavioral Disorders, 15, 3-20.
Martin, R. P., Hooper, S., & Snow, J. (1986). Behavior rating scale approaches to personality assessment in children and adolescents. In H. Knoff (Ed.), The assessment of child and adolescent personality (pp. 309-351). New York: Guilford. Mash, E. J., & Dozois, D. J. A. (2002). Child psychopathology: A developmental systems perspective. In E. J. Mash & R. A. Barkley (Eds.), Child psychology (2nd ed., pp. 3-71). New York: Guilford.
McKinney, J. D., Montague, M., & Howcutt, A. M. (1998). Systematic screening children at risk for developing SED: Initial results from a prevention project. ERIC Document Reproduction Service No. ED432855.
Merrell, K. W. (2003). Behavioral, social, and emotional assessment of children and adolescents (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
Murray, C., & Greenberg, M. T. (2006). Examining the importance of social relationships and social contexts in the lives of children with high-incidence disabilities. Journal of Special Education, 39, 220-233.
Nelson, J. R., Benner, G. J., Lane, K., & Smith, B. W. (2004). Academic achievement of K-12 students with emotional and behavioral disorders. Exceptional Children, 71, 59-73.
Nelson, J. R., Benner, G. J., Reid, R. C., Epstein, M. H., & Currin, D. (2002). The convergent validity of office discipline referrals with the CBCL-TRF. Journal of Emotional and Behavioral Disorders, 10, 181-189.
Presidents Commission on Excellence in Special Education. (2002). A new era: Revitalizing special education for children and their families. Retrieved September 5, 2007, from http://www.ed.gov/inits/ commissionsboards/whspecialeducation/reports/index.htm]
Severson, H. H., Walker, H. W, Hope-Doolittle, J., Kratochwill, T. R., & Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45, 193-223.
Soodak, L., Podell, D., & Lehman, L. (1998). Teacher, student, and school attributes as predictors of teachers' responses to inclusion. Journal of Special Education, 31, 480-497.
Sugai, G., & Horner, R. (1999). Discipline and behavior support: Practices, promise, and pitfalls. Effective School Practices, 17, 10- 22.
Sugai, G., Horner, R., & Gresham, F. M. (2002). Behaviorally effective school environments. In M. Shinn, H. Walker, & G. Stoner (Eds.), Interventions for academic and behavior problems II: Preventive and remedial approaches (pp. 315-350). Bethesda, MD: National Association of School Psychologists.
U.S. Department of Education. (2001). Twenty-third annual report to Congress on the implementation of the Individuals With Disabilities Education Act. Washington, DC: Author.
Wagner, M., Friend, M., Bursuck, W. D., Kutash, K., Duchnowski, A. J., Sumi, C., & Epstein, M. H. (2005). Educating students with emotional disturbances: A national perspective on programs and services. Journal of Emotional and Behavioral Disorders, 14(1), 12- 30.
Wagner, M., Kutash, K., Duchnowski, A. J., Epstein, M. H., & Sumi, W. C. (2005). The children and youth we serve: A national picture of the characteristics of students with emotional disturbances receiving special education. Journal of Emotional and Behavioral Disorders, 13, 79-96.
Walker, B., Cheney, D., Stage, S., & Blum, C. (2005). Schoolwide screening and positive behavior supports: Identifying and supporting students at risk for school failure. Journal of Positive Behavior Interventions, 7, 194-204.
Walker, H. M., Nishioka, V. M., Zeller, R., Severson, H. H., & Feil, E. G. (2000). Causal factors and potential solutions for the persistent underidentification of students having emotional or behavioral disorders in the context of schooling. Assessment for Effective Intervention, 26(1), 29-39.
Walker, H. M., & Severson, H. H. (1992). Systematic Screening for Behavior Disorders (2nd ed.). Longmont, CO: Sopris West.
Zigmond, N. (2006). Twenty-four months after high school: Paths taken by youth diagnosed with severe emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 14, 99- 107.
Zlomke, L. C., & Spies, R. (1998). Review of the Systematic Screening for Behavior Disorders. In J. C. Impara, B. S. Plake, & L. L. Murphy (Eds.), The thirteenth mental measurements yearbook (pp. 995-996). Lincoln, NE: Buros Institute.
Ellie L. Young
Michael J. Richardson
Benjamin J. Young
K. Richard Young
Brigham Young University, Provo, UT
Authors' Note: This study was funded in part by U.S. Office of Special Education Programs Grant No. H324C030124. Please address correspondence to Paul Caldarella, BYU-PBSI, 236 South 700 East, Provo, UT 84606; e-mail: [email protected]
Paul Caldarella is the director of Brigham Young University Positive Behavior Support Initiative and adjunct associate professor in the counseling psychology and special education department. His research interests include assessment and intervention for youth with emotional and behavioral issues.
Ellie L. Young is an associate professor in the counseling psychology and special education department at Brigham Young University and directs the school psychology program. Her research interests include gender issues and adolescents with emotional and behavioral concerns.
Michael J. Richardson is a research assistant with the Brigham Young University Positive Behavior Support Initiative and a doctoral student in psychology. His research interests include adolescent development and moral education.
Benjamin J. Young is the data manager and technology specialist for the Brigham Young University Positive Behavior Support Initiative. He assists in the conducting of research and is pursuing his master's degree in community counseling.
K. Richard Young is dean of the David O. McKay School of Education at Brigham Young University and professor in the counseling psychology and special education department. His research interests include applied behavior analysis, positive behavior support, and youth with emotional and behavioral issues.
Copyright PRO-ED Journals Jun 2008
(c) 2008 Journal of Emotional and Behavioral Disorders. Provided by ProQuest Information and Learning. All rights Reserved.