Center for Evaluating the Research Pipeline (CERP):

Providing a New Way to Evaluate Diversity Initiatives in Computing Research

By Jane Stout, Director of CERP

 It is now well established that the field of computing research is lacking in demographic diversity, both in the academy and in industry. To address this issue, many computing-related mentorship and training programs with diversity goals have been established. But do they really work? And how, exactly, do we determine whether they do? In this article, I discuss the lack of diversity that exists in computing, examples of programs that have been developed to address the lack of diversity, and a new evaluation center at the CRA headquarters that offers rigorous, comparative evaluation of how participants in a given program fare in their computing career progression relative to non-participants. We invite all computing community members to get involved with CERP by (a) providing data to enable us to do comparative evaluation, (b) employing our infrastructure for program evaluation, and/or (c) by being an active audience as we learn about ways to increase diversity in computing.

Figure 2. Race Distribution in the U.S. Population and Among Undergraduate Computing Majors in a CERP sample.

The imbalance of representation is particularly striking when one considers both gender and racial composition of computing simultaneously. CERP’s sample of undergraduate computing majors in Figure 3 shows that women are underrepresented across racial groups, men compose 77% of the sample, and white men alone make up greater than 50% of the sample.

Figure 3. Gender x Race Make-up of Undergraduate Computing Majors in the CERP Sample.

Why should we care about this lack of diversity in the computing research world? When one demographic group dominates the field, computing lacks a diversity of perspectives and experiences, both of which promote innovation and creativity that are essential to problem solving.  For example, NCWIT has found patents with mixed-gender teams are more highly cited in subsequent patents compared to patents authored by one gender.  The issue of low diversity in computing becomes cyclical when underrepresented individuals feel like outsiders due to the low presence of other people who are “like them.” Social science research indicates that this sense of alienation can lead individuals to avoid or drop out of computing, thus perpetuating underrepresented individuals’ low participation in computing.

It goes without saying that all people, regardless of their gender, race, ethnicity, income level or disability status, should have the opportunity to pursue a computing career, and feel welcomed and efficacious while doing so. A number of faculty members, members of industry, government and policy affiliates, and non-profit professionals are working to create greater diversity in computing by developing mentorship, training, and professional networking programs for underrepresented individuals. For example, CRA-W has developed and runs mentorship programs for women at all stages of their career that aim to enhance the skills, confidence and intrinsic interest needed for those women to advance professionally in computing. There are numerous other programs in place to enhance diversity in computing such as AccessComputing, which focuses on increasing access to computing careers for people with disabilities; iAAMCS, a brand-new mentoring program for enhancing African American individuals’ participation in computer science; MentorNet, which pairs successful professionals and computing students with an eye towards offering mentorship to women and other underrepresented individuals in engineering and related sciences; and CAHSI, an alliance of universities that work to boost the number of Hispanic  individuals who pursue computing degrees. These are but a few of the many existing computing-related mentorship and training programs with diversity goals.

With a continuing growth in interventions aimed at increasing diversity in computing comes a need for rigorous, comparative evaluation of how program participants fare relative to non-participants. In 2010, the CRA-W/CDC Alliance was awarded a Broadening Participation in Computing (BPC) grant from the NSF to pilot the “Data Buddies Project” – an initiative that sought to collect data from a sample of students enrolled in computing programs nationwide with the generous assistance of department chairs and staff. These would serve as comparative data against which one could compare outcomes of intervention participants to outcomes of the “average student”. Data collection was a success. With the Data Buddies framework established, in May of 2012, the CRA-W/CDC applied for an NSF BPC grant that would fund a research center called CRA’s Center for Evaluating the Research Pipeline (CERP) using the Data Buddies model.

In September of 2012, the grant was awarded and CERP is now housed at the CRA office in Washington D.C. As of March 1, 2013, Dr. Jane Stout directs CERP, and is joined by Dr. Jessica Cundiff, who is CERP’s Research Analyst. Both Drs. Stout and Cundiff are social psychologists that study women and underrepresented minority groups’ involvement in STEM fields. Dr. Betsy Bizot, who is CRA’s Director of Evaluation, also collaborates on CERP projects. All members of the CERP team have expertise in program evaluation and quantitative methods. The number of CERP staff continues to grow; we are currently seeking a qualified research assistant who will help perform the rapidly increasing number of administrative tasks necessary for CERP to thrive.

The ultimate goal of CERP is to use its unique Data Buddies infrastructure to facilitate the computing community’s efforts to thicken and build diversity in the research pipeline. One way CERP will do this is through its continued relationship with a national sample of academic computing departments (i.e., our “Buddies”). In this relationship, departments encourage students and faculty to complete Data Buddies surveys. In return, CERP provides each department with an individualized report on how their own students and faculty are progressing in the research pipeline compared to the national sample. To date, many departments have found these reports to be useful for internal evaluation purposes. A second way CERP helps broaden the research pipeline is by using the collected Data Buddies data for comparative evaluation of interventions with diversity initiatives. Specifically, CERP works with “clients” who have interventions in place (e.g., CRA-W/CDC DREU participants), by comparing outcomes of the “average student” with outcomes of students who participated in a client’s program(s). Because CERP staff has extensive behavioral science and quantitative expertise, CERP is well situated to collect and analyze comparative data for our clients. For example, using Data Buddies data, CERP has found that undergraduate students who have participated in CRA-W/CDC REU programs are more likely than students who had no REU experience or other REU experiences to be enrolled in a graduate program in computing the fall after graduation; that graduate program is also more likely to be a PhD degree program for CRA-W/CDC REU participants than all other students (see Figure 4).

Figure 4. CRA-W/CDC REU Students and all Other Students’ Enrollment in Computing Graduate Programs After College. Note: Majority male students include Asian and White men. Underrepresented minorities and women (URM + W) include American Indian, Black, Hawaiian or Hispanic men and women of all ethnic backgrounds. Other research experiences include research projects at one's home institution during the school year as well as summer research projects at another university. CRA-W/CDC REU programs included Collaborative Research Experience for Undergraduates (CREU), Distributed Research Experience for Undergraduates (DREU) and Multidisciplinary Research Opportunities for Women (MRO-W).

CERP has a number of ambitious goals for the coming years. This summer, CERP will build its network of Data Buddies by recruiting more participating departments to complete the Data Buddies surveys. Increasing participation will help ensure that CERP consistently obtains large and demographically diverse samples of students and faculty members in our comparative datasets.  Ultimately, we also aim to display our comparative data using an interactive web interface. For example, viewers could use such an interface to observe patterns of persistence in computing among demographic groups of interest. Most immediately, we are seeking a stable constituency of CERP clients who need CERP’s comparative evaluation structure and analytic rigor. Through all of these initiatives, it is our goal to become a sustainable, national resource for the computing research community.  Join us in diversifying computing.