Abstract:
Numerous ratings systems of higher educational institutions exist and inform the decisions
of many students each year. However, none of these academic ranking systems have been shown
to be statistically valid, which raises the question of whether the information students are using is
accurate. The major challenge in the construction of valid predictive models for ranking academic
systems is the fact that “academic quality” is not well defined. The only known resolution to this
problem is through the use of reputational surveys (surveys of people in academics deemed
qualified to judge an academic system’s quality). Although this approach is based on the well
accepted method of academic peer review, evidence suggests that reputational ratings of academic
programs are biased to some degree by the reviewer’s perception of the program’s home institution
and not based solely on the quality of the program itself (the so called “halo effect”). In this thesis,
a method for testing the validity of reputational surveys as a measure of academic quality is
presented and applied to the U.S. News & World Report rankings of U.S. doctoral programs in
statistics. Our initial results suggest that there is insufficient evidence to conclude reputational
surveys provide a valid measure of educational quality.
