Remediation in the Community College: An Evaluator’s Perspective
By Levin, Henry M Calcagno, Juan Carlos
Remediation is the most common approach to preparing students academically and socially during their early stages of college. However, despite its profound importance and its significant costs, there is very little rigorous research analyzing its effectiveness. The goal of this article is to provide a conceptual framework for the evaluation of remedial education programs. Based on previous literature, we review a list of ingredients for successful interventions, present a number of approaches to remediation that make use of these ingredients, discuss alternative research designs for systematic evaluations, and enumerate basic data requirements. Keywords: community colleges; program evaluation; remedial education
The “remediation crisis” has surely become one of the most controversial issues in higher education in recent times.1 Large numbers of students accepted into colleges and universities are underprepared for the content and rigor of coursework at this level. And much of the underpreparation involves academic skills that are foundational to learning, such as those used in mathematics, reading, and writing. The demand for remedial courses has increased rapidly in recent decades, especially at community colleges, which have opened their doors to all students whatever their level of academic preparedness (Dougherty, 1994, 2003). More than 60% of first-time community college students in the National Education Longitudinal Study of 1988 (NELS: 88) took at least one remedial course, compared to 29% of first-time students in public 4-year institutions (Bailey, Jenkins, & Leinbach, 2005).
Despite the need, providing remedial courses is costly to students, institutions, and governments. These courses are costly to students, because they usually do not confer college credit; thus, students must pay fees and tuition and support themselves without earning credit for a degree. As a result, some students are discouraged from enrolling in the first place, and others fail to complete remedial courses in which they enroll (Deil-Amen & Rosenbaum, 2002; Levin & Koski, 1998; Rosenbaum, 2001). As for institutions, they spend large amounts of resources on remediation and other programs designed to make up for the deficiencies of their diverse entering students. A decade ago Breneman and Haarlow (1998) estimated that public colleges spent between US$1 billion and $2 billion annually on remedial education programs. More recently, a report from the Florida Legislature found that remediation at the Florida community colleges in 2004-2005 cost $118.3 million, 53% of which was paid by the state (Office of Program Policy and Government Accountability, 2006). The portion paid by the state, $62.9 million, represented 4.5% of the 2004-2005 Florida community college operating budget of $1.39 billion (Florida Community College System …, n.d.). It is not surprising to note that state legislatures, which often pay for remediation, question the need to pay twice for academic preparation in the same skills (Merisotis & Phipps, 2000).
The ongoing debate about remediation continues without a useful knowledge base that could inform policy makers, educators, scholars, and students about the effectiveness of different approaches to remediation. As many researchers have already pointed out, the majority of evaluations of remedial education have serious methodological flaws (Bailey & Alfonso, 2005; Grubb, 2001). The goal of tins article is to provide a conceptual framework for the evaluation of various remedial education programs. Based on the previous literature, we review a list of ingredients for successful interventions, present a number of approaches to remediation that make use of these ingredients, discuss alternative research designs for systematic quantitative evaluations, and enumerate basic data requirements.
Situating Postsecondary Remediation
Grubb and Associates (1999) defined remediation as “a class or activity intended to meet the needs of students who initially do not have the skills, experience or orientation necessary to perform at a level that the institutions or instructors recognize as ‘regular’ for those students” (p. 174). These courses have been a prominent feature in community colleges since these institutions first appeared in postsecondary education in the early years of the 20th century (Cohen & Brawer, 2003). However, community college students who are referred to remedial coursework make up a very diverse group. They vary from students who have done poorly in high school in all subjects to students who are deficient in just a single subject. Many are older students who performed satisfactorily in their high school studies but who have rusty skills because of disuse. Others have very poor study habits or have mild to serious learning problems that must be addressed. Finally, many community colleges have significant immigrant populations comprising students who may possess the underlying academic skills for college-level work but who have difficulty with English. This tremendous variety of student types suggests that long-term solutions must be diverse.
Institutions identify such students either by administering placement tests in basic skills or by reviewing grades and prior courses listed on high school transcripts. For example, many states contract with the College Board to implement ACCUPLACER tests, a computer-adaptive placement testing system designed to facilitate the evaluation and placement of college students in three basic skills areas: reading, writing, and mathematics. It is especially noteworthy that the placement tests that are used to identify students for remediation are usually calibrated to select students who have severe deficiencies, typically those who lack the skills required at the eighth grade. In most cases, remedial courses are mandatory for students identified by placement tests; in other cases, advisors and counselors recommend or encourage students to attend remedial courses, however the courses are not required. Even where there is a presumed requirement, however, students and faculty members may still find many ways to avoid remediation. For example, Perin (2006) reported that despite state or institutional mandatory assessment policies, not all students and skills are always assessed, or sometimes instructors may override the assessment requirements.
What are the costs of remediation? First, there are the direct costs of providing remedial instruction and the duplication of effort in using higher educational institutions to provide instruction on subjects that should presumably have been learned earlier. According to the most recent national estimates available, the total annual cost of remedial courses across all types of higher education institutions was between $1 billion and $2 billion some 10 years ago (Breneman & Haarlow, 1998). However, a recent report by the Ohio Board of Regents found that remedial courses are less expensive than college-level courses (Glenn & Wagner, 2006). Moreover, remediation at 2-year institutions is less costly than at 4-year colleges, because many remedial courses have large class sizes and are taught by low-paid adjunct (part-time) faculty members (Bettinger & Long, 2007).
Critics of remediation point out, however, that there are many hidden costs as well, such as the dilution of rigor in regular college-level courses as more and more remedial students must be accommodated (Bennett, 1994; MacDonald, 1998). Steinberg (1998) cited examples of regular course offerings that were once viewed as being remedial; he also cited examples of full-year courses that covered what was once included in one-semester courses. Costrell (1998) also pointed to the fact that the presence of large numbers of remedial students places pressure on instructors to reduce course content and raise grades, diluting the quality of instruction for nonremedial students. What is more, in terms of monetary costs, Breneman and Haarlow’s (1998) estimate did not take account of the foregone earnings that students lose during the period of remediation, even though this is considered a major cost of higher education by economists. Finally, others have argued that remediation carries substantial social costs due to the lower completion rates of remedial students (Glenn & Wagner, 2006).
The costs and the large numbers of students enrolled in remedial courses have generated a debate about where remediation should take place. At least 10 states prevent or discourage public 4-year institutions from offering remedial education (Jenkins & Boswell, 2002), whereas many states restrict remediation to 2-year institutions. As mentioned above, it is often argued that shifting courses to community colleges can result in substantial savings because it is less costly to remediate students in 2-year institutions than in 4-year colleges. However, opponents argue that community colleges are poorly equipped and inadequately funded to deal with the least well prepared students (Astin, 2000). Moreover, it is argued tiiat if community colleges unwittingly hinder the chances of receiving a bachelor’s degree, then this trend toward basing remedial education at 2-year colleges will reduce the educational opportunities of minority, immigrant, and low-income students who are disproportionately less well prepared for postsecondary education (Shaw, 1997). Given the costs of these programs and the controversies discussed above, there is also a question of whether successful completion of remedial courses ensures that students can succeed in the college mainstream. Using NELS: 88 data, Attewell, Lavin, Domina, and Levey (2006) found that about 70% of students pass the reading and writing remedial courses they enroll in; however, only 30% pass all of their remedial matiiematics courses. (This proportion actually overstates the success among the initial population assigned to remediation because it does not include students who were discouraged by their assignment to remedial courses and thus dropped out prior to taking such courses.) However, the degree to which remedial courses improve students’ chances of academic success is almost unknown because of a lack of rigorous follow-up studies. It is not possible to evaluate the effectiveness of remedial courses and practices without a rigorous evaluation design that accounts for student proficiencies and other characteristics. In fact, the literature provides little definitive evidence of the effectiveness of remedial courses and practices on such outcomes as persistence to graduation, quality of performance in subsequent courses, and grade point average (GPA).
Choosing Remedial Interventions
If there is any consensus among educators concerning remediation, it is that so-called drill-and-skill approaches are falling out of favor. Yet, though there is no reliable national survey of the teaching techniques used in remedial courses at community colleges, casual observation at many sites suggests that drill-and-skill approaches are still dominant (e.g., Grubb & Associates, 1999). Such courses are based on the presentation of concepts, operations, or classification schemes, and they employ repetitive practice (often in learning laboratories) to master what is being taught. This style of pedagogy has many drawbacks, including the fact that many remedial students face serious attitudinal obstacles that prevent them from learning in this way. Often it is the same style that the students were exposed to in high school and that may have contributed to their difficulties in the first place. Beyond that, its abstract and isolated nature may prevent students from seeing the usefulness of what is being taught in real-world situations and from applying the skills that are learned to later academic and vocational coursework.
Based on the previous literature on remediation in higher education and adult learning, Levin and Koski (1998) found the following ingredients to be central for designing successful interventions for underprepared students in higher education:2
* motivation: building on the interests and goals of the students and providing institutional credit toward degrees or certificates;
* substance: building skills within a substantive or real-world context as opposed to using a more abstract approach;
* inquiry: developing students’ inquiry and research skills to help them investigate other subjects and areas about which they might be curious;
* independence: encouraging students to do independent meandering within the course structure so that they will develop their own ideas, applications, and understandings;
* multiple approaches: using collaboration and teamwork, technology, tutoring, and independent investigation as suited to student needs;
* high standards: setting high standards and expectations that all students will meet if they exert adequate effort and if they are given appropriate resources to support their learning;
* problem solving: viewing learning less as an encyclopedic endeavor and more as a way of determining what needs to be learned and how to develop a strategy that will succeed;
* connectiveness: emphasizing the links among different subjects and experiences, and showing how they can contribute to learning, rather than seeing each subject and learning experience as isolated and independent;
* supportive context: recognizing that to a large degree learning is a social activity that thrives on healthy social interaction, encouragement, and support.
A number of approaches that build on these ingredients have shown some success relative to isolated courses or study skills. We have grouped these practices into three categories: (a) restructuring the curriculum, (b) developing new institutional structures, and (c) employing specific instructional strategies or technologies that are designed to enhance learning. These categories are not mutually exclusive, and each can be (and is) used to supplement and complement the others.
Restructured Curriculum
A large number of remedial education experts and practitioners recommend that basic skills be taught in conjunction with content course materials so that students gain experience in transferring these skills to tasks that are perceived to be “real” (Blanc, DeBuhr, & Martin, 1983; Commander & Smith, 1995; Stone & Jacobs, 2006; Wilcox, delMas, Stewart, Johnson, & Ghere, 1997). They believe that skills taught in isolation are less likely to be applied productively to further coursework. The obvious solution is to tie basic skills development to concrete applications in academic or vocational courses.
There are many models for doing this, including adjunct courses, tandem classes, paired courses, packaged courses, linked courses, and, in a variant form, supplemental instruction and learning communities. As an overall description, we will call all of these approaches linked courses. At their very essence, linked courses offer instruction in a basic skills subject, such as writing or study skills, to students who are simultaneously registered in a credit-bearing content course, such as history or science. Materials in the content course are used for instruction in the remedial course. The content course may be one of the student’s own selection, or it may be tied to a specific lower division course requirement.
Supplemental instruction (SI) and the learning community are variants of the linked-courses model. Although SI is decidedly not a remedial program and does not link a specific skill-building course to the remedial course, it embeds effective learning and study strategies in a course that supplements the remedial course (Arendale, 2005; Martin & Arendale, 1994).3 This supplemental course is most often taught by a trained student who has already succeeded in the remedial course. The learning-community approach links courses for a group of coregistrants and provides other supportive academic and social services to students. Thus it combines a new approach to curriculum with larger institutional changes, as discussed below.
Developing New Institutional Structures
The learning community is one approach to remediation that shows particular promise based on results from experimental (Brock & LeBlanc, 2005; Tinto, 1997) and quasi-experimental (Wilcox et al., 1997) evaluations.4 It is built on the well-established finding that persistence and success in higher education depend not only on the quality of instruction but also on the integration of students into the social and academic life of the institution (Tinto, 1993). In this approach, groups of courses are taken together by the same cohort of students with the purpose of establishing student communities around learning. At the same time, these communities are provided with other forms of support that integrate the social and academic sides of college participation and engage students more fully in the life of the institution. Examples of such supports are college orientation courses or “student success” classes that focus on learning styles, study skills, time management, and successful habits.5 Learning communities usually also include a restructured curriculum and other features, such as student-faculty collaboration.
In addition to the remedial courses themselves, many community colleges have created learning assistance centers, which are quite diverse in the types of services they offer. In general, these centers are independent of traditional academic departments and holistic in their approach to student development. Such services are open to all students; however, they tend to focus particular attention on the needs of remedial students (Perin, 2004). Nevertheless, because they are designed to meet the needs of all students who desire some extra assistance, the impact of such support services on remedial students is unclear. Typical services include career counseling, peer and faculty tutoring, group tutoring, self-guided computerbased instruction, study skills classes, and additional diagnostic testing.
Modified Classroom Strategies
Modifying what goes on in the classroom-that is, adopting alternative instructional strategies and technologies-is another approach to remediation. These strategies include meaning-centered methods in reading remediation and collaborative learning. Such strategies are based on knowledge of how young adults and adult learners are best taught. Some strategies also use technology and computers to deliver basic skills. Many institutions have implemented self-directed computer-assisted modules that instruct underprepared students in their areas of weakness and that also provide diagnostic feedback and monitoring of progress on a highly individualized basis.
Finally, some educators advocate instruction in higher order thinking skills across the curriculum, including the remedial curriculum (Chaffee, 1992, 2004).6 Critical thinking, complex problem solving, and abstract reasoning have long been the hallmarks of programs aimed at the academically gifted; however, such skills are traditionally considered not within the realm of immediate possibility for most remedial students, who are assumed to need basic skills instruction first. Some educators have challenged this notion, however, and have asserted that instruction in critical thinking can benefit all students, including remedial students. This limited discussion of approaches to remediation illustrates the plethora of possibilities available in the education of remedial students. Often different approaches to remediation are utilized within the same institution, varying from subject to subject and classroom to classroom according to the inventiveness of individual staff members. Although costs and effectiveness can be compared among interventions to see if there are clear distinctions among them in terms of productivity, rarely are these different approaches subject to rigorous evaluations that ascertain their effects. This raises a dilemma for community colleges when it comes to determining which strategies to adopt. To improve performance in preparing remedial students for college-level courses and to compare outcomes across interventions, an institution might wish to answer a number of questions:
1. What are the background characteristics of students taking remedial courses?
2. What proportion of the students who are required to take remedial courses actually enroll in and pass the courses, and how many attempts are needed to pass the courses?
3. What levels of proficiency are exhibited by students who pass courses in each remedial subject relative to nonremedial students?
4. What kinds of courses are undertaken in the areas of initial weakness and with what results?
5. What are the completion rates in subsequent courses and the baccalaureate transfer rates for students who were required to take remedial courses, and what is the typical length of time to graduation in comparison with students who did not take remedial courses?
6. What is the effect of institutional factors, such as the percentage of faculty members who are part-time or the availability of professional development for faculty members, on the effectiveness of remedial courses?
7. What are the best approaches for addressing the needs of an institution’s underprepared students? How can its faculty and staff understand the impacts of specific interventions?
Answering these questions will require a systematic evaluation of each intervention and background data on students to control statistically for other differences between remedial and nonremedial students. Both of these components are analyzed below.
Evaluating Interventions: Methodological Problems and a Hierarchy of Solutions
We begin with the truism that the consequences of an intervention cannot be known without systematic evaluation. The costs and educational outcomes of alternative interventions need to be ascertained to choose wisely among different approaches. Light, Singer, and Willett (1990) showed that many outcomes can be assessed through systematic evaluation. This section discusses the empirical problems associated with evaluating the effectiveness of remedial interventions and suggests a range of rigorous evaluation designs. The estimation of costs is not discussed here but can be determined through a careful cost analysis as discussed in Levin and McEwan (2001).
The lack of high-quality research evidence on the effectiveness of remediation is due to the paucity of available data and to the abundant methodological flaws in what has been attempted with existing data. Methodological flaws in previous studies have been pointed out by several researchers. For example, Grubb (2001) argued that “many existing evaluations are useless because they fail to recognize what the program does – and therefore they provide little information about what should be changed to make it more effective” (p. 4). Just as important, previous empirical research does not address the issue of how and why students enter remedial programs in the first place (Bailey & Alfonso, 2005).
The main statistical problem in estimating the effectiveness of remedial courses is that it is difficult to identify a causal relationship between remediation and educational attainment. Students are not randomly assigned to remedial education; therefore, factors unobserved by the statistician may also influence future outcomes of remedial students. Thus, if we simply compare the performances of remedial versus nonremedial students in terms of educational outcomes, the former group will perform far worse than the latter group due mainly to precollege differences rather than to the program itself (Bettinger & Long, 2005a; Grubb, 2001). We should, instead, compare only those remedial and nonremedial students who actually share similar backgrounds and academic preparedness. By doing so, the effects of an intervention can be attributed to the program rather than to precollege differences.
A related problem is that evaluations of later academic success typically do not take into account the fact that those who successfully complete remedial courses will do better than those who drop out. A generally accepted finding in the literature is that students who successfully complete remedial courses have better educational outcomes than similar nonremedial students (e.g., Attewell et al., 2006). However, the group of completers of remedial coursework is in fact a self-selected sample of all remedial students. Therefore, a comparison between successful remediation completers and nonremedial students biases the results of the intervention upward because the group of low-performing students who dropped out without completing the remedial sequences is excluded from the analysis. If students are discouraged from enrolling in remediation in the first place, or if they enroll but fail to complete remedial courses, this discouragement effect should be taken into account in the evaluation.
Methodological flaws abound in previous studies. Even so, the hierarchy of methods to obtain causal inferences is widely known by quantitatively oriented applied researchers. The “gold standard” for evaluation is random assignment, followed by quasi-experiments, and, finally, nonexperimental designs (Shadish, Cook, & Campbell, 2002). This hierarchy has recently been endorsed by the U.S. Department of Education through several technical reports sponsored by the Institute of Education Sciences and the What Works Clearinghouse (Coalition for Evidence-Based Policy, 2005; Myers & Dynarski, 2003; What Works Clearinghouse, 2006). In the next sections, we discuss each of these research designs with examples of recent applications used to evaluate remedial education programs. Depending on the resources and data available, these designs can be used for evaluation at the institutional, state, or national levels. Likewise, all of these designs can be used either to estimate the efficacy of broadly defined remedial programs or to evaluate specific practices, such as those discussed in the previous section.
Experimental Designs
Experimental designs based on random assignment of students to treatments provide the most credible evidence of the consequences of an intervention.7 Essentially, when implemented carefully, random assignment leads to the creation of two virtually identical groups at baseline, with the only difference being that only one group (the program group) is exposed to the intervention, whereas the other group (the control group) is not. A control group generated through random assignment provides the best way of describing what would have happened to students in a treatment group if they had not been exposed to the treatment (Holland, 1986; Rubin, 1974). Hence, any changes observed between the two groups over time can be attributed to the effects of the intervention with a known degree of statistical precision. Baseline measures of student proficiency in the subject as well as attitudinal variables are taken to ensure that randomization has produced equivalent groups. The equivalency of the two groups can be confirmed by measuring and comparing motivation, attitude toward the subject, high school courses and grades, demographic variables (age, gender, race, socioeconomic status), English language proficiency, living conditions, and family and work obligations.
At the beginning and end of the intervention, proficiency tests in the subject and attitudinal measures toward it are administered to compare gains in proficiency and attitudinal orientations. In addition, postintervention measures are made of student completions, attendance, and pass rates. Comparisons are also made of attrition rates, the timing of attrition, and the types of students who experience attrition. Statistical adjustments are sometimes made for differential attrition if the experimental group (those students exposed to the intervention) and control group (those students who proceed in the traditional fashion) end up with different compositions of students. Presumably, the difference in performance on educational outcome measures can be attributed to the intervention relative to the traditional treatment. These are short- term results in that there is no assurance that they will be maintained over the long run, so longer-run outcomes, such as persistence toward degree completion and performance in higher level courses, must be examined at a later time.
Of course, the internal validity of this design is enhanced if, when one (experimental) version of a remedial course is being compared to its traditional counterpart, the same instructor is used for both. This may be impractical, however, if a single instructor is not equally enthusiastic or adept at both approaches. Alternatively, when two different instructors are selected, it is desirable to assign instructors with comparable past success in teaching similar students. Past student evaluations might be used to make such a selection. Also, an attempt must be made to avoid Hawthorne and John Henry effects. Hawthorne effects are created when students perceive that they are receiving special treatment or status by virtue of being chosen for the intervention. The conferment of special status in the experimental period may stimulate higher motivation and achievement; however, the same effects may not be present when the intervention is replicated and routinized under more conventional conditions. The John Henry effect occurs when the instructor or the students in the traditional course exert a special level of effort to show that they can beat the alternative intervention, even though this effort is unlikely to be marshaled or sustained under routine implementation. Presence of the Hawthorne effect overstates the efficacy of the intervention when replicated routinely; presence of the John Henry effect understates it. It is also wise to document the intervention carefully. This can be accomplished by using qualitative methods to observe and describe activities and interactions among students as well as the interactions between students and the instructor. Reading materials and assignments can be evaluated for their content and the demands that they place on students. Periodic measures of attendance, hours devoted to preparing for class, and student attitudes toward the course are valuable indicators of student behavior. This information can be used later to analyze the actual differences (as opposed to just the differences in design) between the experimental and the traditional courses with a possibility of identifying salient features that contribute to success.
Random assignment evaluations of remediation are scarce. When they do occur, they are often carried out to evaluate learning communities. For example, a comprehensive evaluation of a learning communities program and its cost is being conducted at Kingsborough Community College as part of the Opening Doors Demonstration (Brock & LeBlanc, 2005). Incoming first-year students who meet certain eligibility criteria were randomly assigned either to the learning communities program or to a control group that participates in the standard course format.8 The intervention for the experimental group consisted of three classes (a college orientation, an English class, and a standard college course) that groups of approximately 25 students take together as a block during their first semester. Opening Doors also includes a voucher that covers the cost of the books, and students have access to a dedicated tutor who can assist them with English and other course assignments (D. Bloom & Sommo, 2005). Preliminary findings suggest that incoming first-year students in the learning communities program passed remedial English courses at higher rates and earned more course credits overall after 1 year of follow-up than students in the control group (D. Bloom & Sommo, 2005). It is important to note that students assigned to the learning communities were also more than twice as likely to pass the English examination required by the City University of New York for graduation or for transfer to a 4-year school.
True experiments such as the one just described are costly and very uncommon in education. Among other reasons for this, Cook (2002) argued that philosophical and practical reasons undermine the potential benefits of experiments in education. Besides the fact that institutions rarely have the capacity to undertake experimentation of this sort, there is also the practical problem of having enough students taking particular courses at precisely the same time to assign them randomly. However, the latter challenge can be overcome in many community colleges because of the very large number of students who must take remediation. Multiple sections can be established that meet at precisely the same time, and students who register for the course can be assigned randomly to specific sections. It can be argued that there is no ethical dilemma here in terms of assigning some students to “better” instruction and others to “poorer” instruction as long as there is no history of definitive advantages for either alternative.9
Quasi-Experimental Designs
Random assignment is considered the “gold standard” for evaluation; however, random assignment must be carried out correctly and with adequate numbers of students to ensure the similarity of comparison groups and the power to detect policy-relevant impacts (Light et al., 1990; Riecken & Boruch, 1974). Also, though random assignment can help in improving internal validity, it may also reduce external validity-the generalizability of results to other settings. Often, the only realistic alternative to random assignment is a quasi-experimental design, which attempts to simulate experimental studies through statistical controls (Shadish et al., 2002). By definition, quasi-experiments lack random assignment; however, researchers often have control over design elements, such as the covariates that are used to assign study subjects to treatment and that allow researchers to construct useful counterfactual inferences. Shadish et al. (2002) considered that regression-discontinuity, interrupted time-series, and matched cohort designs are the strongest alternatives to randomized experiments. Nevertheless, such quasi-experimental designs ought to include a careful analysis to identify and reduce the possibility of alternative causal explanations.
Research on remedial education can take advantage of assignment policies as sources of quasi-experimental design. As examples, we review here four quasi-experimental strategies that use assignment policies to evaluate the effectiveness of remedial programs and that can be used at the state or institutional level: (a) regression- discontinuity designs using cutoff assignment policies; (b) instrumental variable approaches that can be employed when institutions within a state use the same standardized test to assess student skills but employ different cutoff scores to assign students to remedial courses; (c) matched cohorts, which can be employed when students are referred to remediation but are not mandated to take remedial courses; and (d) strategies that employ changes in assignment policies. All of these designs can be used either to estimate the efficacy of broadly defined remedial programs or to evaluate specific practices such as those discussed above.
Regression-discontinuity designs. One common strategy at the institutional or state level is to use standardized placement tests and cutoff policies to assign students to remedial classes. For example, all first-time-in-college, degree-seeking students entering a community college or state university in Florida must demonstrate certain basic skills before beginning college-level courses (Windham, 2005). Basic skills are measured using standardized test scores, and minimum score requirements are set statewide and established by the State Board of Education. Incoming students who do not achieve minimum scores in each subject area on the exams must take remedial classes before they begin college-level work in each subject. In other words, students are assigned to either remedial or college-level courses, depending on their scores on the standardized tests, not by coin toss or lottery as in a randomized experiment.
Mandatory cutoffs create a sharp discontinuity in the probability of receiving remedial classes that can be exploited for evaluation. Students with test scores above a cutoff are not assigned to remediation, and those scoring below are referred to remedial classes. A regression-discontinuity design uses a basic linear regression framework for a given outcome (e.g., passing the first college-level course) and employs a remediation indicator and the test score, or any other variable used for assignment, as independent variables.10 This natural experiment allows a comparison between two observationally alike students, one receiving remediation and the other not receiving remediation, who differ only in that one scored just below the cutoff score and the other just above it. After including the test score in the statistical model, assignment to remediation is not correlated with the error term. (Such correlation is the fundamental cause of distortions due to selection bias discussed above.) The regression coefficient can be interpreted as the causal impact of the intervention for students on the margin of passing the cutoff, and the results are thus comparable to a randomized experiment (Rubin, 1977; Trochim, 1984).11
One example of this regression-discontinuity evaluation design is provided by Calcagno (2007), who took advantage of the statewide cutoff policy in Florida to estimate the causal impact of remedial education on students’ educational outcomes. Results suggest a positive effect of remediation on the likelihood of enrolling in the following fall term for students on the margin of passing the cutoff. The author found no significant difference between remedial and nonremedial students on the margin of the cutoff score in terms of passing the first college-level courses, earning an associate’s degree, or transferring to a state 4-year college. The remedial program appeared to increase the time needed to achieve these educational outcomes for students in remedial mathematics as compared with their academically equivalent peers. However, there was no statistical evidence that remediation extends time-to-degree for academically equivalent students who received an associate’s degree in Florida. This same research design could be used in other states using statewide cutoff policies, such as Texas and Maryland, or in individual institutions when cutoffs are known.12
Instrumental variable approach. In other states, such as Ohio, Connecticut, and Virginia, the state authorities encourage institutions to use particular standardized placement tests; however, the assignment rules are left to the individual institutions. For these cases, researchers can combine the different institutional cutoff policies and the distance from students’ homes to higher education institutions to estimate the effectiveness of remedial programs at the state level. Previous research has shown that students are more likely to attend one college over another depending on how close the colleges are to their homes (Rouse, 1995); however, given betweencollege variation in remediation placement policies, the distance from home also affects the probability of receiving remediation for different students. Therefore, this situation generates a natural experiment that allows researchers to compare two observationally alike students, one assigned to remedial courses and the other not assigned to remedial courses. In practice, the analytical technique for estimating the program effect in this setting is an instrumental variable approach.13 The “instrument” is created by combining two-step regressions. In the first step, researchers estimate the probability of assignment to remediation given the test scores and the different cutoff policies used by different institutions. The second stage estimates the likelihood of attending college X given the distance from student’s home to each college. The output of these regression analyses generates a combined variable as the product between the likelihood of a student choosing a given institution and the likelihood of being placed into remediation at that college. This instrument does influence the assignment to remediation; however, it does not have an impact on outcomes (a condition also known as the exclusion restriction).
Bettinger and Long (2005a) used this strategy with data from public 4-year institutions in Ohio. They found that remediation in mathematics and English reduced the likelihood of dropping out and increased the likelihood of completing a degree. Remedial students taking English classes were also less likely to transfer to a less selective or lower level college in comparison to similar nonremedial students. The same authors extended their research to compare the outcomes of similar students taking the ACT and those who enrolled in 2-year institutions in Ohio (Bettinger & Long, 2005b); they found that students placed in remedial mathematics courses were 15% more likely to transfer to a 4-year college than students with similar test scores and high school preparation who attended colleges with policies that did not require placement in remedial classes. These students were also more likely than their natural comparison group to take 10 more credits hours. Results for remediation in English were not statistically significant across all of the outcomes.
Matched cohorts. Some states require institutions to assess the academic preparedness of students and place them accordingly, but without specifying a standardized placement test or cutoff scores. Under this policy incentive, institutions rely heavily on faculty perceptions of need for remediation. Jepsen (2006) exploited this flexibility to estimate the impact of remedial classes in California community colleges (CCC). Basic-skills enrollment is not mandatory in die CCC system. Faculty members encourage students to enroll in remedial classes based on multiple assessments; however, students can decide whether or not to enroll. Hence, the author matched students into pairs that were considered similar in terms of preparation for college-level work by the faculty or staff; each matched pair included one student who decided to enroll in remedial classes and another who did not. Data problems prevented the author from calculating statewide effects; however, preliminary results for 12 institutions suggested that remedial classes were positively associated with second-term continuation in college and with the completion a transfer-level class. For younger students, the classes were negatively related to the probability of transferring, whereas for older students the classes were positively related to the probability of receiving a degree or certificate.
Changes in assignment policies. Finally, other sources for quasiexperimental designs are policy changes at the institutional, state, or national level. Lavin, Alba, and Silberstein (1981) provided an institutionallevel example of the possible use of this evaluation design. The authors conducted a meticulous evaluation by taking advantage of the change in admission policies implemented by the City University of New York in 1970. Given the new open- enrollment policy at the time, more underprepared students decided to enroll at the university; however, the institution was not able to provide remediation for all of the students who scored low on the placement test, nor was it able to introduce mandatory remedial classes for them. As a result, the authors were able to identify a group of comparable students who were candidates for remediation but who differed in that some elected to take the remedial courses whereas some did not. After controlling statistically for differences in high school background and level of need for remediation, they found that success in remedial courses increased the probability that students would return for a second year by about 7% to 8%, and that the probability of graduation or transfer to a senior institution increased by about 2% to 3%; all results were relative to similar students not taking the remedial courses. Paradoxically, students taking remedial courses did not have better GPAs than similar students who did not take remedial courses. However, students not taking remedial courses may have had other means (e.g., tutoring) of obtaining academic help, something that was not measured in this study.
Nonexperimental Designs and Mixed Methods
A nonexperimental design does not include random assignment components or any “controllable” design elements that could allow researchers to generate natural counterfactuals. This approach is the weakest of the evaluation designs, with wide variance in application and potential validity. When employing this design, researchers generally use statistical techniques to select a comparison group based on observable pretreatment differences. Researchers often try to eliminate selection bias through a statistical assumption-that by including control variables (e.g., prior academic results and socioeconomic status), selection to remedial courses can be ignored.14 It is unfortunate to note that the assumption becomes difficult to defend in the context of higher education because key factors, such as motivation or ability, are difficult to measure. The internal validity of the results depends on the quality of the techniques and their capacity to control for unobservable characteristics.
Nonexperimental designs used to evaluate the effectiveness of remedial courses should be conducted only when key student characteristics such as socioeconomic background, test scores, and data on previous educational experiences are well measured.15 A careful nonexperimental evaluation at the national level was conducted by Attewell et al. (2006) using data from NELS: 88. The authors used propensity score matching and key student background characteristics to provide a detailed picture of the effectiveness of remedial courses.16 Their results suggest that, on average, remediation had no negative effect on degree completion at 2-year colleges (although results depended on the subject of the course), but that there was a negative effect of 6% to 7% on degree completion at 4-year colleges (especially for reading). Yet they also found evidence that community college students who successfully completed remedial courses had better educational outcomes than similar nonremedial students, a result that has been consistently supported in previous research (Lavin et al., 1981; Moss & Yeaton, 2006).
Finally, it is also possible to use mixed methods in conducting research by combining quantitative and qualitative approaches. For example, researchers may first use one of the previously discussed quantitative methods to rank institutions according to the effectiveness of remedial classes. Then, an arbitrary number of top- and low-ranking institutions could be selected for field research. Qualitative methods might then be used for an in-depth study of policies and practices that could help explain the differences in educational outcomes of remedial and nonremedial students at different institutions.
Data Sources and Requirements
Sources of data for experimental designs generally include surveys and academic records over several years, and these data are usually collected specifically for a particular study. Most of the quantitative information used in quasi-experimental and nonexperimental designs, however, is regularly collected by states or institutions through administrative data sets, as illustrated in the cases of Florida, Ohio, and California. The educational outcomes and control variables that are chosen for use in such a study depend on the specifics of the student populations and the goals of the intervention. For example, in the case of a course that is designed to improve writing and reasoning skills, the following might be viewed as potential variables for short-term educational outcomes: course completion, grade, writing performance, reasoning performance, attendance, and course satisfaction. Long-term outcomes might include first college-level course enrollment, completion rates in those first college-level courses, fall-to-fall retention, certificate or degree completion, and transfer to baccalaureate- granting institutions.
If randomization is properly achieved and the experimental design minimizes the problems discussed above, then comparing average outcomes for treatment and control groups using standard statistical tests should suffice to estimate program impacts. In other cases, an attempt should be made to gather not only baseline data on each student’s performance but also demographic data and data on previous educational performance to control statistically for preintervention differences between treatment and control groups. For experimental designs, and certainly for quasi-experimental and nonexperimental designs, researchers might gather information for each student using die following variables: baseline writing performance, baseline reasoning performance, gender, race, native language, age, socioeconomic status, high school courses completed (specific courses or General Equivalency Diploma [GED]), GPA earned in pertinent high school courses, work hours per week, hours devoted to household demands (e.g., the care of others), and word-processing skills and access to a computer. These control variables can be used in multivariate analyses to adjust outcomes statistically for differences between students who are included in different interventions (Shadish et al., 2002). Data on costs are also important. Interventions have different costs attached to them, and these need to be compared, along with effectiveness, to ascertain if an intervention under study is “worth it.” Levin and McEwan (2001) provided the analytical tools for a careful cost analysis of remedial education interventions. Different costs may be associated with various interventions because of differences in the use of adjunct versus full-time faculty members or because of limits on class size or on other resources, such as the use of counselors or extra faculty time, that are required for some interventions. These can be taken into account when comparing intervention alternatives. In some cases, more expensive interventions may be more cost- effective when one considers that costs associated with students who leave the community college without completing even the most basic requirements represent squandered resources for the college, the student, and society. If a higher cost remedial intervention substantially increases the number of students who complete basic requirements and graduate or transfer to 4-year institutions, the additional cost may be more than compensated for by a reduction in poorly spent resources.
Discussion
The view taken in this article is that the number of underprepared students at community colleges and 4-year institutions has become alarmingly high. In the long run, vast improvements in elementary and secondary education will be needed to raise the quality of preparation for postsecondary education. However, for the foreseeable future, higher education institutions in the United States will be faced with the formidable challenge of assisting large numbers of underprepared students succeed in postsecondary education.
In this respect, we make two central assertions. First, the traditional drill-and-skill approach that is often used to raise the performance of remedial students at community colleges is not as productive as other available alternatives. A number of remedial approaches discussed here have shown some success, especially in terms of reducing dropout and failure rates. second, community colleges need to carry out formal evaluations of different remedial approaches to test their efficacy and cost-effectiveness in order to pursue a wise remediation strategy. The research designs described above can be used in these evaluations.
Previously, Levin (1991) contended that higher education institutions need to become “experimenting institutions” if they are to continually improve their productivity. They need to constantly seek better ways of meeting their objectives, testing and replicating them with an eye toward the cost-effectiveness of alternatives. Why does this ideal seem far removed from the actual practices of many institutions, and, in particular, community colleges? Experimenting institutions that build “local” knowledge used for policy decisions must have clear goals, incentives to reach those goals, information on present performance, and information on the likely consequences of alternatives. Yet institutional research carried out at most community colleges is not aligned with these elements. Instead, institutional research largely entails the gathering of information for simple reports of cross-sectional data required by federal and state governments and accreditation agencies. In most cases the institutional research function is staffed by only a single professional with limited clerical support, and in some cases, that professional lacks training and experience in evaluation (Morest & Jenkins, 2007). In short, there exists little real capacity to carry out rigorous research at most community colleges.
These institutional conditions do not augur well for experimentation and systematic evaluation. Yet some steps could be taken that would assist community colleges. It probably makes sense to establish a central resource at the state level and cooperative efforts with universities to assist community colleges and individual faculty members in creating experimental interventions and to provide support for evaluating them. Standard intervention designs and data collection centers could be established as well as methods for analyzing data on outcomes and costs. Faculty members and administrators could collaborate with the evaluation staff inside or outside of their institutions to specify the appropriate outcomes and control variables, help administer the data instruments, and assist in the interpretation of results. Implementing steps such as these could encourage improvements in the evaluation of various educational practices and programs, including remedial interventions.
Authors’ Note: Funding for this study was provided by Lumina Foundation for Education through the Achieving the Dream: Community Colleges Count initiative (For more information see http:// www.achievingthedream.org). We are grateful to Estela Bensimon, Davis Jenkins, Doug Slater, the editor, and referees for their comments on an earlier draft and for detailed comments and suggestions that have improved the article.
Notes
1. In many quarters the term remedial education has been replaced with the term developmental education. Here, however, the terms remediation, college-prep, and developmental education, are used interchangeably.
2. Many of these ingredients have been applied successfully in the Accelerated Schools Project in which so-called at-risk students are provided with enrichment experiences, a pedagogy associated with gifted and talented instruction instead of remediation (Finnan & Swanson, 2000; Levin, 1997). See H. Bloom et al. (2001) for evaluation results.
3. In a seminal article, Blanc, DeBuhr, and Martin (1983) described an early supplemental instruction (SI) program and reported the evaluation results.
4. For reviews of learning communities, see Taylor and Associates (2003) and Price (2005).
5. See Florida Department of Education (2006) and Zeidenberg, Jenkins, and Calcagno (2007) for nonexperimental evaluation results of the effects of “student life skills” (SLS) programs on community college student success.
6. The Critical Thinking Program at LaGuardia Community College was evaluated using a variety of strategies by Garlie Forehand of the Educational Testing Service (ETS). See Chaffee (1985).
7. A good source on random assignment and experimental design is Riecken and Boruch (1974).
8. In the Opening Doors Demonstration, randomization is present among the eligible group. Still, participation is voluntary, and some noncompliers can be expected. For an excellent description of the analytical methods for estimating the impact of experimental designs when there are noncompliers, see Appendix A in Brock and LeBlanc (2005).
9. With respect to the ethical issue of assigning students to better or poorer instruction, institutions seem to have no inhibitions in assigning students to instructors who are known to differ markedly in the quality of their teaching. It is noteworthy that higher standards are set for experimentation than for routine instruction.
10. This research design maintains its properties when multiple instruments are used to assign students to remedial classes, as long as all measures of the instruments are included in the regression.
11. A straightforward instrumental variable solution is available when institutions do not comply with statewide cutoff policies. The same approach is used in experimental designs when participation is voluntary and some noncompliers are expected. See Appendix A in Brock and LeBlanc (2005). See also Lesik (2006), and Calcagno (2007).
12. For examples of applications at the institutional level, see Lesik (2006) and Moss and Yeaton (2006).
13. A good source on instrumental variables techniques is Wooldridge (2003).
14. This statistical assumption is called “selection on observables” or the “ignorability condition.”
15. The best known nonexperimental evaluation of remedial education programs and their program characteristics was conducted in the early 1990s by the National Center for Developmental Education (Boylan, Bliss, & Bonham, 1992, 1994, 1997). However, only brief summaries have been reported for this major national study, and the reports exclude information that would be helpful for evaluating the effectiveness of remediation (Bailey & Alfonso, 2005).
16. See H. Bloom (2005) for a good summary on properties of propensity score matching and its controversies.
References
Arendale, D. (Comp.). (2005). Postsecondary peer cooperative learning programs: Annotated bibliography. Retrieved October 20, 2006, from http://www.tc.umn.edu/~arend011/Peerbib03.pdf.
Astin, A. (2000). An interview: Alexander W. Astin. National Crosstalk, 8(1), pp. 2-3.
Attewell, P., Lavin, D., Domina, T., & Levey, T (2006). New evidence on college remediation. Journal of Higher Education, 77, 886-924.
Bailey, T, & Alfonso, M. (2005). Paths to persistence: An analysis of research on program effectiveness at community colleges. Indianapolis, IN: Lumina Foundation for Education. Bailey, T, Jenkins, D., & Leinbach, T (2005). Community college low-income and minority student completion study: Descriptive statistics from the 1992 high school cohort. New York: Columbia University, Teachers College, Community College Research Center.
Bennett, W. (1994). The devaluing of America. New York: Touchstone.
Bettinger, E., & Long, B. (2005a). Addressing the needs of under- prepared college students: Does college remediation work? Cambridge, MA: National Bureau of Economic Research.
Bettinger, E, & Long, B. (2005b). Remediation at the community college: Student participation and outcomes. In C. A. Kozeracki (Ed.), Responding to the challenges of developmental education (New Directions for Community Colleges, No. 129, pp. 17-26). San Francisco: Jossey-Bass.
Bettinger, E., & Long, B. (2007). Institutional responses to reduce inequalities in college outcomes: Remedial and developmental courses in higher education. In S. Dickert-Conlin & R. Rubenstein (Eds.), Economic inequality and higher education: Access, persistence and success (pp. 69-100). New York: Russell Sage Foundation.
Blanc, R., DeBuhr, L., & Martin, D. (1983). Breaking the attrition cycle: The effects of supplemental instruction on undergraduate performance and attrition. Journal of Higher Education, 54, 80-90.
Bloom, D., & Sommo, C. (2005). Building learning communities: Early results from the opening doors demonstration at Kingsborough Community College. New York: MDRC.
Bloom, H. (Ed.). (2005). Learning more from social experiments: Evolving analytic approaches. New York: Russell Sage Foundation.
Bloom, H., Rock, J., Ham, S., Mielton, L., & O’Brien, J. (with F. Doolittle & S. Kgahiro). (2001). Evaluating the accelerated schools approach. New York: MDRC.
Boylan, H., Bliss, L., & Bonham, D. (1992). The impact of developmental education. Review of Research in Developmental Education, 9(5), 1-3.
Boylan, H., Bliss, L., & Bonham, D. (1994). Characteristic components of developmental programs. Review of Research in Developmental Education, 11(1), 1-4.
Boylan, H., Bliss, L., & Bonham, D. (1997). Program components and their relationship to students’ performance. Journal of Development Education, 20(3), 2-8.
Breneman, D., & Haarlow, W. (1998). Remedial education: Costs and consequences. Fordham Report, 2(9), 1-22.
Brock, T., & LeBlanc, A. (2005). Promoting student success in community college and beyond: The Opening Doors demonstration. New York: MDRC.
Calcagno, J. C. (2007). Evaluating the impact of developmental education in community colleges: A quasi-experimental regression- discontinuity design. Unpublished doctoral dissertation, Columbia University, Teachers College, New York.
Chaffee, J. (1985). Final report: Critical thinking atLaGuardia Community College. Washington, DC: National Endowment for the Humanities.
Chaffee, J. (1992). Critical thinking skills: The cornerstone of developmental education. Journal of Developmental Education, 15(3), 2-8, 39.
Chaffee, J. (2004). Thinking critically (8th ed.). Boston: Houghton-Mifflin.
Coalition for Evidence-Based Policy. (2005). Reporting the results of your study: A user-friendly guide for evaluators of educational programs and practices. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
Cohen, A. M”& Brawer, F B. (2003). The American community college (4th ed.). San Francisco: Jossey-Bass.
Commander, N., & Smith, B. (1995). Developing adjunct reading and learning courses that work. Journal of Reading, 38, 352-360.
Cook, T (2002). Randomized experiments in education: Why are they so rare? Educational Evaluation and Policy Analysis, 24, 175-199.
Costrell, R. (1998). Commentary. Fordham Report, 2(9), 23-40.
Deil-Amen, R., & Rosenbaum, J. (2002). The unintended consequences of stigma-free remediation. Sociology of Education, 75, 249-268.
Dougherty, K. (1994). The contradictory college: The conflicting origins, impacts, and futures of the community college. Albany: State University of New York Press.
Dougherty, K. (2003). The evolving role of the community college. In J. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 17, pp. 295-348). New York: Agathon Press.
Finnan, C, & Swanson, J. (2000). Accelerating the learning of all students: Cultivating culture change in schools, classrooms, and individuals. Boulder, CO: Westview.
Florida Community College System community college operating budgets with 2007-08 state appropriations, (n.d.). Retrieved May 16, 2007, from http://www.fldoe.org/cc/newsletter/ pdf/ Operating_Budgets.pdf.
Florida Department of Education. (2006, November). Taking student life skills courses increases academic success (Data Trend 31). Retrieved September 24, 2007, from http://www.fldoe.org/ cc/ gohigher/2007_03_12/Data_ Trend_31.pdf.
Glenn, D., & Wagner, W. (2006, June). Cost and consequences of remedial course enrollment in Ohio public higher education: Six- year outcomes for fall 1998 cohort. Paper presented at the Association of Institutional Research Forum, Chicago.
Grubb, W. N. (2001). From black box to Pandora’s box: Evaluating remedial/developmental education. New York: Columbia University, Teacher’s College, Community College Research Center.
Grubb, W. N., & Associates. (1999). Honored but invisible: An inside look at teaching in community colleges. New York: Routledge.
Holland, P. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945-970.
Jenkins, D., & Boswell, K. (2002). State policies on community college remedial education: Findings from a national survey. Denver, CO: Education Commission of the States.
Jepsen, C. (2006, April). Basic skills in California’s community colleges: Evidence from staff and self referrals. Paper presented at the American Educational Research Association Meeting, San Francisco, CA.
Lavin, D., Alba, R., & Silberstein, R. (1