COWLES FOUNDATION FOR RESEARCH IN
ECONOMICS Box 208281
COWLES FOUNDATION DISCUSSION PAPER NO. 1833 Meritocracy Voting: Measuring the Unmeasurable Peter C.B. Phillips October 2011 Learned societies commonly carry out selection processes to add new
fellows to an existing fellowship. Criteria vary across societies but are typically based
on subjective judgments concerning the merit of individuals who are nominated for
fellowships. These subjective assessments may be made by existing fellows as they vote in
elections to determine the new fellows or they may be decided by a selection committee of
fellows and officers of the society who determine merit after reviewing nominations and
written assessments. Human judgment inevitably plays a central role in these
determinations and, notwithstanding its limitations, is usually regarded as being a
necessary ingredient in making an overall assessment of qualifications for fellowship. The
present paper suggests a mechanism by which these merit assessments may be complemented
with a quantitative rule that incorporates both subjective and objective elements. The
goal of 'measuring merit' may be elusive but quantitative assessment rules can help to
widen the effective electorate (for instance, by including the decisions of editors, the
judgments of independent referees, and received opinion about research) and mitigate
distortions that can arise from cluster effects, invisible college coalition voting and
inner sanctum bias. The rule considered here is designed to assist the selection process
by explicitly taking into account subjective assessments of individual candidates for
election as well as direct quantitative measures of quality obtained from bibliometric
data. The methodology has application to a wide arena of quality assessment and
professional ranking exercises. |
