The validity of reputation survey based academic ranking systems for U.S. doctoral programs

Numerous ratings systems of higher educational institutions exist and inform the decisions of many students each year. However, none of these academic ranking systems have been shown to be statistically valid, which raises the question of whether the information students are using is accurate. The major challenge in the construction of valid predictive models for ranking academic systems is the fact that “academic quality” is not well defined. The only known resolution to this problem is through the use of reputational surveys (surveys of people in academics deemed qualified to judge an academic system’s quality). Although this approach is based on the well accepted method of academic peer review, evidence suggests that reputational ratings of academic programs are biased to some degree by the reviewer’s perception of the program’s home institution and not based solely on the quality of the program itself (the so called “halo effect”). In this thesis, a method for testing the validity of reputational surveys as a measure of academic quality is presented and applied to the U.S. News & World Report rankings of U.S. doctoral programs in statistics. Our initial results suggest that there is insufficient evidence to conclude reputational surveys provide a valid measure of educational quality.