Are Female Applicants Disadvantaged in National Institutes of Health Peer Review? Combining Algorithmic Text Mining and Qualitative Methods to Detect Evaluative Differences in R01 Reviewers' Critiques
Publication: Journal of Women's Health
Volume 26, Issue Number 5
Abstract
Background: Women are less successful than men in renewing R01 grants from the National Institutes of Health. Continuing to probe text mining as a tool to identify gender bias in peer review, we used algorithmic text mining and qualitative analysis to examine a sample of critiques from men's and women's R01 renewal applications previously analyzed by counting and comparing word categories.
Methods: We analyzed 241 critiques from 79 Summary Statements for 51 R01 renewals awarded to 45 investigators (64% male, 89% white, 80% PhD) at the University of Wisconsin-Madison between 2010 and 2014. We used latent Dirichlet allocation to discover evaluative “topics” (i.e., words that co-occur with high probability). We then qualitatively examined the context in which evaluative words occurred for male and female investigators. We also examined sex differences in assigned scores controlling for investigator productivity.
Results: Text analysis results showed that male investigators were described as “leaders” and “pioneers” in their “fields,” with “highly innovative” and “highly significant research.” By comparison, female investigators were characterized as having “expertise” and working in “excellent” environments. Applications from men received significantly better priority, approach, and significance scores, which could not be accounted for by differences in productivity.
Conclusions: Results confirm our previous analyses suggesting that gender stereotypes operate in R01 grant peer review. Reviewers may more easily view male than female investigators as scientific leaders with significant and innovative research, and score their applications more competitively. Such implicit bias may contribute to sex differences in award rates for R01 renewals.
Get full access to this article
View all available purchase options and get full access to this article.
References
1.
Committee on Maximizing the Potential of Women in Academic Science EU, Committee on Science, & Public Policy (US). Beyond bias and barriers: fulfilling the potential of women in academic science and engineering. Washington, DC: National Academy Press, 2007.
2.
Herring C. Does diversity pay?: Race, gender, and the business case for diversity. Am Sociol Rev 2009;74:208–224.
3.
Hong L, Page SE. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc Natl Acad Sci U S A 2004;101:16385–16389.
4.
Desvaux G, Devillard-Hoellinger S, Baumgarten P. Women matter: Gender diversity, a corporate performance driver. Paris: McKinsey, 2007.
5.
Carnes M, Morrissey C, Geller SE. Women's health and women's leadership in academic medicine: Hitting the same glass ceiling? J Womens Health 2008;17:1453–1462.
6.
Ruzek SB, Becker J. The women's health movement in the United States: From grass-roots activism to professional agendas. J Amer Med Women's Assoc 1999;54:4–8.
7.
Nicolette J, Jacobs MB. Integration of women's health into an internal medicine core curriculum for medical students. Acad Med 2000;75:1061–1065.
8.
Weiss LB, Levison SP. Tools for integrating women's health into medical education: Clinical cases and concept mapping. Acad Med 2000;75:1081–1086.
9.
Rosser SV. Women's health—Missing from US medicine. Washington, DC: Georgetown University Press, 1994.
10.
Kaatz A, Dattalo M, Regner C, Filut A, Carnes M. Patterns of feedback on the bridge to independence: A qualitative thematic analysis of NIH mentored career development award application critiques. J Womens Health 2016;25:78–90.
11.
Bickel J, Wara D, Atkinson BF, et al. Increasing women's leadership in academic medicine: Report of the AAMC Project Implementation Committee. Acad Med 2002;77:1043–1061.
12.
Wright AL, Schwindt LA, Bassford TL, et al. Gender differences in academic advancement: Patterns, causes, and potential solutions in one US College of Medicine. Acad Med 2003;78:500–508.
13.
Carr PL, Gunn CM, Kaplan SA, Raj A, Freund KM. Inadequate progress for women in academic medicine: Findings from the national faculty study. J Womens Health 2015;24:190–199.
14.
Hofler L, Hacker MR, Dodge LE, Ricciotti HA. Subspecialty and gender of obstetrics and gynecology faculty in department-based leadership roles. Obstet Gynecol 2015;125:471–476.
15.
Hofler LG, Hacker MR, Dodge LE, Schutzberg R, Ricciotti HA. Comparison of women in department leadership in obstetrics and gynecology with those in other specialties. Obstet Gynecol 2016;127:442–447.
16.
Jena AB, Khullar D, Ho O, Olenski AR, Blumenthal DM. Sex differences in academic rank in US medical schools in 2014. JAMA 2015;314:1149–1158.
17.
Burden M, Frank MG, Keniston A, et al. Gender disparities in leadership and scholarly productivity of academic hospitalists. J Hosp Med 2015;10:481–485.
18.
Lautenberger DM, Dandar VM, Raezer CL, Sloane RA. The State of Women in Academic Medicine: The Pipeline and Pathways to Leadership. Washington, DC: Association of American Medical Colleges, 2014.
19.
Association of American Medical Colleges (AAMC). Medical Students, Selected Years, 1965–2013. Available at: www.aamc.org/download/411782/data/2014_table 1.pdf Accessed May 26, 2016.
20.
Svider PF, Mauro KM, Sanghvi S, Setzen M, Baredes S, Eloy JA. Is NIH funding predictive of greater research productivity and impact among academic otolaryngologists? Laryngoscope 2013;123:118–122.
21.
King A, Sharma-Crawford I, Shaaban AF, et al. The pediatric surgeon's road to research independence: Utility of mentor-based National Institutes of Health grants. J Surg Res 2013;184:66–70.
22.
Pohlhaus JR, Jiang H, Wagner RM, Schaffer WT, Pinn VW. Sex differences in application, success, and funding rates for NIH extramural programs. Acad Med 2011;86:759–767.
23.
NIH Data Book—U.S. Department of Health and Human Services. R01-Equivalent grants: Success rates, by gender and type of application. Available at: https://report.nih.gov/nihdatabook/ Accessed March 9, 2016.
24.
Tabak LA, Collins FS. Weaving a richer tapestry in biomedical science: NIH leadership discusses the need for renewed efforts to increase diversity in the US biomedical research workforce. Science (New York, NY) 2011;333:940–941.
25.
Valantine HA, Collins FS. National Institutes of Health addresses the science of diversity. Proc Natl Acad Sci 2015;112:12240–12242.
26.
Carnes M, Geller S, Fine E, Sheridan J, Handelsman J. NIH directors pioneer awards: Could the selection process be biased against women? J Womens Health 2005;14:684–691.
27.
Carnes M, Bland C. Viewpoint: A challenge to academic health centers and the National Institutes of Health to prevent unintended gender bias in the selection of clinical and translational science award leaders. Acad Med 2007;82:202–206.
28.
Carli LL, Alawa L, Lee Y, Zhao B, Kim E. Stereotypes about gender and science: Women ≠ scientists. Psychol Women Q 2016;40:1–17.
29.
Nosek BA, Smyth FL, Sriram N, et al. National differences in gender–science stereotypes predict national sex differences in science and math achievement. Proceed Natl Acad Sci 2009;106:10593–10597.
30.
Leslie S-J, Cimpian A, Meyer M, Freeland E. Expectations of brilliance underlie gender distributions across academic disciplines. Science 2015;347:262–265.
31.
Meyer M, Cimpian A, Leslie S-J. Women are underrepresented in fields where success is believed to require brilliance. Front Psychol 2015;6:235.
32.
Biernat M, Tocci M, Williams JC. The language of performance evaluations gender-based shifts in content and consistency of judgment. Soc Psychol Person Sci 2012;3:186–192.
33.
Isaac C, Chertoff J, Lee B, Carnes M. Do students' and authors' genders affect evaluations? A linguistic analysis of Medical Student Performance Evaluations. Acad Med 2011;86:59–66.
34.
Axelson RD, Solow CM, Ferguson KJ, Cohen MB. Assessing implicit gender bias in medical student performance evaluations. Eval Health Prof 2010;33:365–385.
35.
Trix F, Psenka C. Exploring the color of glass: Letters of recommendation for female and male medical faculty. Discourse Soc 2003;14:191–220.
36.
Kaatz A, Magua W, Zimmerman DR, Carnes M. A quantitative linguistic analysis of National Institutes of Health R01 application critiques from investigators at one institution. Acad Med 2015;90:69–75.
37.
Kaatz A, Lee YG, Potvien A, et al. Application critiques, impact and criteria scores: Does the sex of the principal investigator make a difference? Acad Med 2016;91:1080–1088.
38.
National Institutes of Health: Side-by-side comparison of enhanced and former review criteria (research grants and cooperative agreements). Available at: http://grants.nih.gov/grants/peer/guidelines_general/comparison_of_review_criteria.pdf Accessed April 10, 2016.
39.
National Institutes of Health: Enhancing peer review at NIH. Available at: http://enhancing-peer-review.nih.gov/index.html Accessed April 10, 2016.
40.
Pennebaker JW, Boyd RL, Jordan K, Blackburn K. The development and psychometric properties of LIWC2015. Austin, TX: University of Texas at Austin, 2015.
41.
Aggarwal CC, Zhai C. Mining text data. New York, NY: Springer Science & Business Media, 2012.
42.
Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. J Machine Learn Res 2003;3:993–1022.
43.
Dreyfus HL. What computers still can't do: a critique of artificial reason. Cambridge, MA: MIT Press, 1992.
44.
Bellotti V, Edwards K. Intelligibility and accountability: Human considerations in context-aware systems. Hum Comput Interact 2001;16:193–212.
45.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3:77–101.
46.
National Institutes of Health: Frequently Asked Questions. NIH new and early stage investigator (ESI) policies. Available at: http://grants.nih.gov/grants/new_investigators/investigator_policies_faqs.htm Accessed December 11, 2016.
47.
Office of Extramural Research. National Institutes of Health: NIH Research Project Grant Program (R01). Available at: https://grants.nih.gov/grants/funding/r01.htm Accessed April 9, 2016.
48.
Office of Extramural Research. National Institutes of Health: Peer Review Process. Available at: https://grants.nih.gov/grants/peer_review_process.htm Accessed April 9, 2016.
49.
R Core Team: R: A language and environment for statistical computing. Vienna, Austria: R Core Team. R Foundation for Statistical Computing, 2014.
50.
Hirsch J. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 2005;102:16569–16572.
51.
Hutchins BI, Yuan X, Anderson JM, Santangelo GM. Relative citation ratio (RCR): A new metric that uses citation rates to measure influence at the article level. PLoS Biol 2016;14:e1002541.
52.
McCallum AK. MALLET: A machine learning for language toolkit, 2002. http://mallet.cs.umass.edu Accessed February 14, 2017.
53.
Mimno D. Vignette-mallet: A wrapper around the Java machine learning tool MALLET. 2013. https://CRAN.R-project.org/package=mallet Accessed February 14, 2017.
54.
Ponweiser M. Latent Dirichlet allocation in R, 2–21. Austria: Institute for Statistics and Mathematics, WU (Wirtschaftsuniversitat Wien), 2012.
55.
Chang J, Gerrish S, Wang C, Boyd-Graber JL, Blei DM. Reading tea leaves: How humans interpret topic models. Adv Neural Inf Process Syst 2009;31:288–296.
56.
De Villiers MR, De Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach 2005;27:639–643.
57.
Gupta UG, Clarke RE. Theory and applications of the Delphi technique: A bibliography (1975–1994). Technol Forecast Soc Change 1996;53:185–211.
58.
Hsu C, Sandford B. The Delphi technique: Making sense of consensus. Pract Assess Res Eval 2007;12:1–8.
59.
Landeta J, Barrutia J, Lertxundi A. Hybrid Delphi: A methodology to facilitate contribution from experts in professional contexts. Technol Forecast Social Change 2011;78:1629–1641.
60.
Wickham H. Stringr: Make it easier to work with strings. R package version 0.6. 2012;2:96–7.
61.
Feinerer I, Hornik K. tm: Text mining package. R Package, 2014.
62.
Meyer D, Hornik K, Feinerer I. Text mining infrastructure in R. J Stat Software 2008;25:1–54.
63.
Walker A. openxlsx: Read, write and edit XLSX Files. R Package, 2015.
64.
NVivo qualitative data analysis Software. Victoria, Australia: QSR International Pty Ltd, 2002.
65.
Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods 2006;5:80–92.
66.
Boyatzis RE. Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage, 1998.
67.
Eagly AH, Karau SJ. Role congruity theory of prejudice toward female leaders. Psychol Rev 2002;109:573–598.
68.
Biernat M, Kobrynowicz D. Gender-and race-based standards of competence: Lower minimum standards but higher ability standards for devalued groups. J Pers Soc Psychol 1997;72:544–557.
69.
Ridgeway CL. Gender, status, and leadership. J Soc Issues 2001;57:637–655.
70.
Rudman LA, Kilianski SE. Implicit and explicit attitudes toward female authority. Pers Soc Psychol Bull 2000;26:1315–1328.
71.
Marchant A, Bhattacharya A, Carnes M. Can the language of tenure criteria influence women's academic advancement? J Womens Health 2007;16:998–1003.
72.
Heilman ME, Haynes MC. Subjectivity in the appraisal process: A facilitator of gender bias in work settings. In: Borgida E, Fiske ST, eds. Beyond common sense: Psychological science in the courtroom. Malden, MA: Blackwell Publishing, 2008:127–155.
73.
National Institutes of Health (NIH). Office of Extramural Research. Grants and funding: Definitions of criteria and considerations for research project grant (RPG/X01/R01/R03/R21/R33/R34) critiques. Available at: http://grants.nih.gov/grants/peer/critiques/rpg_D.htm-rpg_overall Accessed April 13, 2016.
74.
Jackson D, Engstrom E, Emmers-Sommer T. Think leader, think male and female: Sex vs. seating arrangement as leadership cues. Sex roles 2007;57:713–723.
75.
Porter N, Geis FL, Jennings J. Are women invisible as leaders? Sex Roles 1983;9:1035–1049.
76.
Watts JH. ‘Now you see me, now you don't’: The visibility paradox for women in a male-dominated profession. Revealing and concealing gender. New York, NY: Springer, 2010:175–193.
77.
Biernat M. Stereotypes and shifting standards: Forming, communicating, and translating person impressions. In: Devine PGP, E.A., ed. Advances in Experimental Social Psychology. San Diego, CA: Academic Press, 2012;45:1.
78.
Uhlmann EL, Cohen GL. Constructed criteria redefining merit to justify discrimination. Psychol Sci 2005;16:474–480.
79.
Eagly AH, Wood W, Diekman AB. Social role theory of sex differences and similarities: A current appraisal. In: Eckes T, Trautner HM eds. The Developmental Social Psychology of Gender. Mahwah, NJ: Lawrence Erlbaum Associates, 2000:123–174.
80.
Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J. Science faculty's subtle gender biases favor male students. Proc Natl Acad Sci 2012;109:16474–16479.
81.
Steinpreis RE, Anders KA, Ritzke D. The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: A national empirical study. Sex Roles 1999;41:509–528.
82.
King EB, Botsford W, Hebl MR, Kazama S, Dawson JF, Perkins A. Benevolent sexism at work gender differences in the distribution of challenging developmental experiences. J Manage 2012;38:1835–1866.
83.
Correll S, Simard C. Vague Feedback Is Holding Women Back. Harvard Business Review. 2016.
84.
Foschi M. Double standards in the evaluation of men and women. Soc Psychol Q 1996;59:237–254.
85.
Castilla EJ. Gender, race, and meritocracy in organizational careers. Am J Sociol 2008;113:1479–1526.
86.
Vescio TK, Gervais SJ, Snyder M, Hoover A. Power and the creation of patronizing environments: The stereotype-based behaviors of the powerful and their effects on female performance in masculine domains. J Person Soc Psychol 2005;88:658–672.
87.
Wilson KY. An analysis of bias in supervisor narrative comments in performance appraisal. Hum Relat 2010;63:1903–1933.
88.
Carnes M, Devine PG, Manwell LB, et al. The effect of an intervention to break the gender bias habit for faculty at one institution: A cluster randomized, controlled trial. Acad Med 2015;90:221–230.
89.
Jagsi R, Guancial EA, Worobey CC, et al. The “gender gap” in authorship of academic medical literature—A 35-year perspective. N Engl J Med 2006;355:281–287.
90.
Cole JR, Zuckerman H. The productivity puzzle: Persistence and change in patterns of publication of men and women scientists. Adv Motiv Achiev 1984;2:217–258.
91.
Toutkoushian RK, Bellas ML. Faculty time allocations and research productivity: Gender, race and family effects. Rev High Educ 1999;22:367–390.
92.
Fox MF. Research, teaching, and publication productivity: Mutuality versus competition in academia. Sociol Educ 1992:293–305.
93.
Stack S. Gender, children and research productivity. Res High Educ 2004;45:891–920.
94.
Settles IH, Cortina LM, Malley J, Stewart AJ. The climate for women in academic science: The good, the bad, and the changeable. Psychol Women Q 2006;30:47–58.
95.
Leahey E. Gender differences in productivity research specialization as a missing link. Gender Soc 2006;20:754–780.
96.
Allison PD, Stewart JA. Productivity differences among scientists: Evidence for accumulative advantage. Am Sociol Rev 1974;39:596–606.
97.
Fox MF. Gender, family characteristics, and publication productivity among scientists. Soc Stud Sci 2005;35:131–150.
98.
Carr PL, Ash AS, Friedman RH, et al. Relation of family responsibilities and gender to the productivity and career satisfaction of medical faculty. Ann Intern Med 1998;129:532–538.
99.
Hunter L, Leahey E. Parenting and research productivity: New evidence and methods. Soc Stud Sci 2010;40:433–451.
100.
Diamond SJ, Thomas CRJ, Desai S, et al. Gender differences in publication productivity, academic rank, and career duration among U.S. academic gastroenterology faculty. Acad Med 2016;91:1158–1163.
Information & Authors
Information
Published In
Copyright
Copyright 2017, Mary Ann Liebert, Inc.
History
Published in print: May 2017
Published online: 1 May 2017
Published ahead of print: 10 March 2017
Topics
Authors
Author Disclosure Statement
No competing financial interests exist.
Ethical Approval
The University of Wisconsin Institutional Review Board approved all aspects of this study. Protocol # SBS2012-1177.
Metrics & Citations
Metrics
Citations
Export Citation
Export citation
Select the format you want to export the citations of this publication.
View Options
Get Access
Access content
To read the fulltext, please use one of the options below to sign in or purchase access.⚠ Society Access
If you are a member of a society that has access to this content please log in via your society website and then return to this publication.