Incentivizing Quality and Impact: Evaluating Scholarship in Hiring, Tenure, and Promotion

Batya Friedman, University of Washington
Fred B. Schneider, Cornell University

Adopted by CRA Board of Directors February 2015

Introduction

A careful distinction between quality and quantity is key to promoting the future growth of the computing and information field. Toward that end, this document advocates adjustments to hiring, promotion, and tenure practices as well as to the publication culture. Contributions in a small number of high quality publications or artifacts are what should be emphasized; success as a researcher is then not primarily a matter of numbers.

The recommendations that follow were developed over an 18-month period by the CRA Committee on Best Practices for Hiring, Promotion, and Scholarship. As part of this work, the committee conducted interviews in autumn 2013 with more than 75 academic and industry computing and information unit heads to understand the issues and gain insights from practice. Preliminary recommendations were vetted with department chairs and CRA Deans at the Snowbird Conference in July 2014.

The Challenges of Success

Computing and Information Science and Engineering research has enjoyed great success over the last four decades. The topics investigated by the discipline have expanded, and the field now touches virtually all aspects of society. Numbers of researchers and the level of activity—in academia and industry—have grown concomitantly. With this growth and success come new challenges, which must be addressed though a thoughtful adjustment in scholarly culture and practices.

The field benefits when researchers build on each other’s work. To do so, requires that research advances be accompanied by discussion of methods, comparisons with related work, inclusion of supporting data and proofs, access to artifacts, and other details. Certain publication formats and review processes, however, encourage practices inconsistent with these elements of good scholarship. Length restrictions often are satisfied by omitting critical content, which hinders reproducing the results, understanding their novelty, or delimiting a contribution’s applicability. The omission of supporting data and proofs, also common practice, hobbles efforts to validate or extend the work.

Surveys and other papers where the primary contribution is to synthesize or reframe existing research can be quite valuable to the field. In a community where highly regarded venues for disseminating research require submissions to be short, pursuing this kind of research is often not favored or popular. This is problematic for the continued health of the field.

Above all, quality and impact need to be incentivized over quantity. Sheer numbers of publications (or derivative bibliometrics) should not be a primary basis for hiring or promotion, because this does not encourage researchers to optimize for quality or impact.  Other proxy measures are similarly problematic. For example, whether program committee service indicates an individual’s stature in the field depends on the conference.  As another example, securing research grants (even when proposals are peer reviewed) does not necessarily signal research quality or even a researcher’s visibility. What ultimately should matter when it comes to hiring, tenure, and promotion is the quality of the research.

Toward Incentivizing Quality and Impact

Hiring, Tenure, and Promotion. Recommendations for hiring, tenure, and promotion practices follow, where some institutional variation may occur.

Hiring Recommendation. Evaluate candidates on the basis of the contributions in their top one or two publications, in concert with the research statement and the other standard material (e.g., letters of recommendation, full CV, teaching statement) generally read by hiring committees in determining whom to invite to campus for an interview and, ultimately, whom to hire. Candidates should identify publications where they have played a significant role.

Tenure and Promotion Recommendation. Evaluate candidates for tenure and promotion on the basis of the contributions in their most important three to five publications (where systems and other artifacts may be included).1 Tenure and promotion committees should invite external reviewers to comment on impact, depth, and scholarship of these publications or artifacts as well as the standard material (e.g., full CV, research statement, teaching statement). Some institutions might ask a candidate to suggest which publications or artifacts be considered, other institutions might leave that determination to the external reviewers. Per standard practice, tenure and promotion committees should read the external letters and the standard material in determining tenure and promotion decisions.

Implementing these recommendations will require attention to the transition for young researchers. Specifically, Ph.D. students and younger researchers should be mentored to focus on producing high quality research, with quantity being a secondary consideration. Annual or reappointment reviews (which often occur after three years of hiring) should reflect the emphasis on quality—not quantity—and should recognize that high caliber research activities may take two or three years to come to fruition (e.g., publication or artifact deployment) and even longer for the impact to become apparent. For these interim evaluations, evidence of steady progress prior to publication may take various forms (e.g., presentation of works-in-progress, revise and resubmit status, working prototypes of systems).

A corollary follows: Evaluation of senior faculty similarly should emphasize quality over quantity, with incentives for pursuing greater risk-taking in research activities.

Publication Culture. Systemic changes throughout the publication culture would help to support better scholarship. With new technology and digital delivery, publishers could remove page limits for reference lists and could allow appendices for data, methods, and proofs. Editors, as appropriate, could consider longer submissions with the understanding that, in such cases, a longer review period would be likely. In addition to conferences with published proceedings, other professional gatherings (that do not publish proceedings) might be held where work-in-progress could be presented. Funding for attending such gatherings should follow normal procedures, with acceptance of a work-in-progress for presentation considered grounds for receiving travel funds. Taken together, these and other like-minded changes work to improve the quality of scholarship.

Managing the Transition

As with any change to a dynamic ecosystem, the computing and information research community needs to be alert to unanticipated side effects, particularly those that might be deemed less desirable. De-emphasizing quantity might predispose researchers to collaborate less often on projects where they do not have the primary role; yet as fields mature, collaborative enterprises historically have become more important. Promotion and tenure committees should be sensitive to an individual who contributes to a range of projects and technical areas, but is not seen as a driving force behind their three to five most important publications. Well-framed incremental research also can be important for advancing the field; opportunities for disseminating this work will continue to be needed. A potential misuse of unlimited reference lists could occur if bibliometrics were to gain purchase with gratuitous citation; this is another reason to approach bibliometrics with caution. De-emphasizing quantity might also discourage risk-taking where explorations would be of great value. Finally, after resetting expectations for an emphasis on quality over quantity, there soon would likely be pressure yet again for increases in numbers (e.g., for two rather than one publication per year for annual review processes). The community must resist this pressure.

Acknowledgments

The CRA Committee on Best Practices for Hiring, Promotion, and Scholarship thanks the academic and industry unit heads who participated in the phone interviews, the participants in the department chairs and CRA Deans feedback sessions at the Snowbird Conference in July 2014, and Janet Davis (Grinnell College) for note-taking during the Snowbird sessions. This work was funded in part by NSF Award No. 1347734.

CRA Committee on Best Practices for Hiring, Promotion, and Scholarship

Members include: Lorenzo Alvisi (University of Texas, Austin), David Culler (University of California, Berkeley), Batya Friedman [Co-chair] (University of Washington), Eric Grimson (Massachusetts Institute of Technology), Mark D. Hill (University of Wisconsin), Julia Hirschberg (Columbia University), Benjamin Kuipers (University of Michigan), Keith Marzullo (National Science Foundation and University of California, San Diego), Tamer Ozsu (University of Waterloo), Frank Pfenning (Carnegie Mellon University), Jennifer Preece (University of Maryland), Fred B. Schneider [Chair] (Cornell University), Eva Tardos (Cornell University), Jennifer Widom (Stanford University), Jeannette Wing (Microsoft Research), and Ellen Zegura (Georgia Tech).

End Notes

[1] For some disciplines represented in Information Schools (e.g., philosophy), the publication outcome is a book, with the expectation that one book would be in press or published at the time of evaluation for tenure.