July 9, 2013
Are The ‘U.S. News’ Medical School Rankings For Primary Care Education Relevant?
UC Davis study questions whether the rankings measure quality
The U.S. News & World Report graduate school rankings inspire discussion and marketing, but do they actually reflect quality differences between schools when it comes to medical education in primary care?
Researchers at the UC Davis School of Medicine are believed to be the first to take on that question by analyzing the primary care rankings from 2009 through 2012, reconstructing scores and proposing alternative measurements. Their findings are published online now, ahead of the August 13 print issue of the journal Academic Medicine.
Printed annually in the Americaâs Best Graduate Schools guidebook, the U.S. News & World Report rankings compare medical school education programs in research, primary care and a few specialties. The goal is to help potential medical students evaluate their options by providing information on the relative quality of the training programs at various schools.
The publicationâs primary care rankings are based on weighted averages of two types of measures: statistical data, including studentsâ MCAT scores and undergraduate GPAs, and expert opinions, including peer assessments from medical school deans and residency program directors.
One of the researchersâ major findings was that the primary care rankings for most of the 89 schools below the top 20 shifted dramatically over the four-year study period. Half of these schools had at least 17 positions separating their best and worst annual ranking in that brief span.
âThese huge shifts in such short timeframes cannot be plausibly attributed to actual changes in quality,â said senior author Anthony Jerant, professor in the UC Davis Department of Family and Community Medicine. Jerantâs collaborators on the research were his UC Davis colleagues Daniel Tancredi, assistant professor of pediatrics, and Klea Bertakis, chair of the Department of Family and Community Medicine.
The team found that small differences in calculations translated into big differences in primary care rankings between schools. In addition, the differences were reported without margins of error showing a schoolâs published ranking and its potentially true position. The authors estimated a typical margin of error to be 22 positions.
âRankings were much more stable for the schools that were consistently in the top 20 during the four-year study period,â said Jerant. âIn fact, the rankings for these schools were so stable it suggested an undue influence of long-standing reputation.â
Also of concern were the scores based on peer surveys of residency program directors and medical school deans.
âIt simply isnât possible for the typical respondent to be highly familiar with the education quality of all of the medical schools they are asked to judge,â said Jerant.
Perhaps in part for this reason, relatively few deans and residency directors respond to U.S. News surveys. During the timeframe of the study, response rates from medical school deans were lower than 50 percent, and response rates from residency directors were below 20 percent. These two components, however, made up 40 percent of a schoolâs final score.
The researchers were also troubled by the âgraduates entering primary careâ rate â or the percentage of graduates who select primary care residencies following medical school â that made up 30 percent of the overall score. U.S. News & World Report assumes that all medical school graduates who select residencies in family medicine, internal medicine or pediatrics end up practicing in one of these primary care fields. Most internal medicine residents and many pediatric residents go on to subspecialty training and never practice primary care, according to Jerant.
Another concern was a measure known as the âfaculty/student ratioâ or the ratio of science and clinical faculty to students. A school might have a large faculty, resulting in a relatively high faculty/student ratio, yet few of the faculty may be involved in primary care education. The researchers also questioned the relevance of the âselectivity rankâ metric, which included mean MCAT scores, undergraduate GPAs and acceptance rates. Some evidence suggests that higher selectivity may work against students interested in primary care, who are more likely to be from under-represented groups and tend to have lower mean standardized testing scores.
The team then experimented with an alternative ranking methodology by removing some of the problematic elements of the U.S. News & World Report methodology, including the âfaculty/student ratioâ and âselectivity rankâ measures. In a second alternate methodology, they also revised the âgraduates entering primary careâ measure, estimating conservatively that half of medical school graduates selecting residencies in internal medicine or pediatrics were destined to practice primary care. These changes produced major differences when compared to the published rankings, as this table from the article shows.
The revised ranking methodologies were not intended to supersede the U.S. News & World Report results but rather to highlight their arbitrariness and sensitivity to changes in key assumptions.
The research teamâs findings also highlight a profound problem â there are no other widely available statistical analyses related to the quality of medical schools.
âThe U.S. News rankings are often considered âgold standardâ measures of how good a medical school is,â said Jerant. âYet there is nothing to compare the rankings against, and they are published without any sort of hedging or acknowledgment that there are limitations to the methodology.â
âWe hope at some point that credentialing organizations will gather and publicize reliable and valid quality information,â Jerant added. âGiven these organizationsâ access to data and history with the nationâs medical schools, they have the best chances for providing accurate and objective comparisons. Until then, prospective medical students interested in careers in primary care should not let the rankings be their sole consideration when judging medical schools.â
A copy of âShort-Term Stability and Spread of the U.S. News & World Report Primary Care Medical School Rankingsâ is available online or by e-mailing Mary Beth de Vilbiss at Academic Medicine at [email protected]
The study was funded in part by a UC Davis Department of Family and Community Medicine Research Grant. The UC Davis School of Medicine is among the nation's leading medical schools, recognized for its research and primary-care programs. The school offers fully accredited master's degree programs in public health and in informatics, and its combined M.D.-Ph.D. program is training the next generation of physician-scientists to conduct high-impact research and translate discoveries into better clinical care. Along with being a recognized leader in medical research, the school is committed to serving underserved communities and advancing rural health. For more information, visit UC Davis School of Medicine.
On the Net: