Health Research Policy and Systems

Health Res Policy Syst. 14: 50

A global call for action to include gender in research impact assessment

Pavel V. Ovseiko

pavel.ovseiko@medsci.ox.ac.uk

, Trisha Greenhalgh

trish.greenhalgh@phc.ox.ac.uk

, Paula Adam

padam@gencat.cat

, Jonathan Grant

jonathan.grant@kcl.ac.uk

, Saba Hinrichs-Krapels

saba.hinrichs@kcl.ac.uk

, Kathryn E. Graham

kathryn.graham@aihealthsolutions.ca

, Pamela A. Valentine

pamela.valentine@aihealthsolutions.ca

, Omar Sued

omar.sued@huesped.org.ar

, Omar F. Boukhris

oboukhris@qnrf.org

, Nada M. Al Olaqi

nolaqi@qf.org.qa

, Idrees S. Al Rahbi

idrees.alrahbi@trc.gov.om

, Anne-Maree Dowd

anne-maree.dowd@csiro.au

, Sara Bice

sara.bice@unimelb.edu.au

, Tamika L. Heiden

theiden@ktaustralia.com

, Michael D. Fischer

michael.fischer@unimelb.edu.au

, Sue Dopson

sue.dopson@sbs.ox.ac.uk

, Robyn Norton

rnorton@georgeinstitute.org

, Alexandra Pollitt

alexandra.pollitt@kcl.ac.uk

, Steven Wooding

wooding@rand.org

, Gert V. Balling

gvb@novo.dk

, Ulla Jakobsen

uj@lundbeckfonden.dk

, Ellen Kuhlmann

kuhlmann@em.uni-frankfurt.de

, Ineke Klinge

i.klinge@maastrichtuniversity.nl

, Linda H. Pololi

lpololi@brandeis.edu

, Reshma Jagsi

rjagsi@med.umich.edu

, Helen Lawton Smith

ubtm188@mail.bbk.ac.uk

, Henry Etzkowitz

h.etzko@googlemail.com

, Mathias W. Nielsen

mwn@stanford.edu

, Carme Carrion

carmecarrion@gmail.com

, Maite Solans‐Domènech

mtsolans@gencat.ca

, Esther Vizcaino

evizcaino@gencat.cat

, Lin Naing

ayub.sadiq@ubd.edu.bn

, Quentin H. N. Cheok

quentin.cheok@ubd.edu.bn

, Baerbel Eckelmann

baerbel@qs.com

, Moses C. Simuyemba

msimuyemba@gmail.com

, Temwa Msiska

temwa.msiska@hotmail.co.uk

, Giovanna Declich

giovanna.declich@asdo-info.org

, Laurel D. Edmunds

laurel.edmunds@medsci.ox.ac.uk

, Vasiliki Kiparoglou

vasiliki.kiparoglou@ouh.nhs.uk

, Alison M. J. Buchan

alison.buchan@utoronto.ca

, Catherine Williamson

catherine.williamson@kcl.ac.uk

, Graham M. Lord

graham.lord@kcl.ac.uk

, Keith M. Channon

keith.channon@cardiov.ox.ac.uk

, Rebecca Surender

rebecca.surender@gtc.ox.ac.uk

, Alastair M. Buchan

alastair.buchan@medsci.ox.ac.uk

Medical Sciences Division, University of Oxford, John Radcliffe Hospital, Oxford, OX3 9DU United Kingdom

Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Primary Care Building, Woodstock Road, Oxford, OX2 6GG United Kingdom

Agency for Health Quality and Assessment of Catalonia (AQuAS), Carrer de Roc Boronat, 81, ES-08005 Barcelona, Spain

The Policy Institute, King’s College London, Strand Campus, London, WC2R 2LS United Kingdom

Alberta Innovates – Health Solutions, 10104-103 Avenue NW, Edmonton, AB T5J 4A7 Canada

Fundación Huésped, Pasaje A. Peluffo 3932 (C1202ABB), Buenos Aires, Argentina

Qatar National Research Fund, P.O. Box 5825, Doha, Qatar

Qatar Foundation, P.O. Box 5825, Doha, Qatar

Department of Studies and Planning, The Research Council, P.O. Box 1422, Al Azaiba, 130 Oman

Commonwealth Scientific and Industrial Research Organisation, P.O. Box 883, Kenmore, Brisbane, 4069 Australia

Melbourne School of Government, The University of Melbourne, Parkville, Victoria 3010 Australia

School of Population Health, University of Western Australia, Perth, WA 6009 Australia

Knowledge Translation Australia Pty Ltd., Melbourne, Victoria Australia

Faculty of Business and Economics, University of Melbourne, 198 Berkeley Street, Parkville, Victoria 3010 Australia

Saïd Business School, University of Oxford, Park End Street, Oxford, OX1 1HR United Kingdom

The George Institute for Global Health, University of Oxford, 34 Broad Street, Oxford, OX1 3BD United Kingdom

The George Institute for Global Health, University of Sydney, P.O. Box M201, Missenden Road, Sydney, NSW 2050 Australia

RAND Europe, Westbrook Centre, Milton Road, Cambridge, CB4 1YG United Kingdom

Novo Nordisk Foundation, Tuborg Havnevej 19, DK-2900 Hellerup, Denmark

Lundbeck Foundation, Scherfigsvej 7, DK-2100 Copenhagen, Denmark

Institute for Economics, Labour and Culture, Goethe-University Frankfurt, Senckenberganlage 31, 60325 Frankfurt am Main, Germany

Medical Management Centre, Department of Learning, Informatics, Management and Ethics (LIME), Karolinska Institutet, Tomtebodavaegen 18a, 171 77 Stockholm, Sweden

Horizon 2020 Advisory Group for Gender, European Commission, Brussels, Belgium

National Initiative on Gender, Culture and Leadership in Medicine: C-Change, Brandeis University Women’s Studies Research Center, 415 South Street, MS 079, Waltham, MA 02454 United States of America

Department of Radiation Oncology, Center for Bioethics and Social Sciences in Medicine, University of Michigan, Ann Arbor, MI 48109 United States of America

Department of Management, Birkbeck, University of London, Malet Street, London, WC1E 7HX United Kingdom

International Triple Helix Institute, 1520 Sand Hill Road, A210, Palo Alto, CA 94304 United States of America

Gendered Innovations, History Department, Stanford University, 450 Serra Mall, Stanford, CA 94305 United States of America

Health Sciences Department, Universitat Oberta de Catalunya, Av. Tibidabo 39-43, ES-08035 Barcelona, Spain

PAPRSB Institute of Health Sciences, Universiti Brunei Darussalam, Jalan Tungku Link, Gadong, BE1410 Brunei Darussalam

Faculty of Integrated Technologies, Universiti Brunei Darussalam, Jalan Tungku Link, Gadong, BE1410 Brunei Darussalam

QS Intelligence Unit, Quacquarelli Symonds Ltd, 4 Heathgate, Agincourt Rd, London, NW3 2NT United Kingdom

Department of Public Health, School of Medicine, University of Zambia, Nationalist Rd, Lusaka, Zambia

Research Support Centre, College of Medicine, University of Malawi, P.O. Box 360, Chichiri, Blantyre 3 Malawi

Assembly of Women for Development and the Struggle against Social Exclusion (ASDO), via Guido Reni 56, 00196 Rome, Italy

NIHR Oxford Biomedical Research Centre, Joint Research Office, Churchill Hospital, Oxford, OX3 7LE United Kingdom

Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, Oxford, OX3 9DU United Kingdom

Department of Physiology, University of Toronto, 1 King’s College Circle, Toronto, Ontario M5S 1A8 Canada

Women’s Health Academic Centre, King’s College London, Guy’s Hospital, London, SE1 1UL United Kingdom

NIHR Biomedical Research Centre at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London, Guy’s Hospital, London, SE1 9RT United Kingdom

Guy’s and St Thomas’ NHS Foundation Trust, Guy’s Hospital, London, SE1 9RT United Kingdom

MRC Centre for Transplantation, King’s College London, Guys’ Hospital, London, SE1 9RT United Kingdom

Department of Social Policy and Intervention, University of Oxford, Barnett House, 32-37 Wellington Square, Oxford, OX1 2ER United Kingdom

Institute of Social and Economic Research, Rhodes University, P.O. Box 94, Grahamstown, 6140 South Africa

© The Author(s). 2016

DOI: 10.1186/s12961-016-0126-z

Published online: 19 July 2016

Abstract

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we – a group of scholars and practitioners from Africa, America, Asia and Europe – argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.

Gender bias in health research

Global investment in biomedical research has grown significantly over recent decades. In 2010, global investment reached US$240 billion (adjusted for purchasing power parity), delivering important health dividends to patients and citizens [1]. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested by either not supporting the best science or by failing to investigate topics that benefit women and men equitably. Gender bias in biomedical and health research involves both biological sex differences and sociocultural differences in the way women and men behave, and in the way they are treated [2]. There is evidence that gender bias in biomedical and health research can occur at all stages of the research process across the following four domains.

First, women tend to be significantly underrepresented in research both as researchers and research participants. Although in 2013, women had reached 55% of admissions to medical schools in the United Kingdom and 47% in the United States of America, they constituted only 28% of faculty physician-scientists in the United Kingdom and 38% in the United States [3]. Moreover, female faculty members tend to report less favourable experiences and feel excluded [4, 5]. Women are also underrepresented as research participants. Historical analysis demonstrated a male bias in biomedical research throughout the 20th century: it was evident in 8 of the 10 biomedical fields surveyed in 2009 [6]. For example, while women represent nearly half of people living with HIV, they are under-represented in clinical studies of HIV antiretroviral drugs (19%), prophylactic vaccines (38%) and curative strategies (11%) [7]. Women are also under-represented in high-impact studies of non-sex-specific cancers [8].

Second, female investigators tend to receive less research funding than their male counterparts in absolute and relative terms. This may occur because there are fewer women investigators who apply for research funding [9], and those who apply receive smaller awards than men [10]. There is some evidence that, in certain settings, this is also amplified by reported nepotism and sexism in peer-review [11]. Although an earlier meta-analysis of empirical studies from different fields concluded that women applying for grants have statistically significant lower odds of receiving funding than men by approximately 7% [12], a more recent and methodologically advanced meta-analysis of the same data [13] and a recent empirical study [14] concluded the contrary. United States research suggests that female early career researchers receive significantly less start-up support from their institutions [15] and are significantly less likely than men to achieve independent funding awards [16]. Further, United States research suggests that women at particular career stages are less likely to apply for the competitive grants for which they are eligible, compared to their male counterparts [17]. Research from the Netherlands demonstrated gender bias favouring male grant funding applicants in the evaluation of the ‘quality of researcher’, but not the ‘quality of proposal’ [18]. Even in the fields where there is no difference in funding rates between the genders, such as radiology, women have less total grant funding than men [19].

Third, women tend to appear less frequently than men as winners of prestigious scientific awards and as authors of research publications. Among 210 Nobel Laureates in Physiology or Medicine awarded from 1901 to 2015, there are only 12 (5.7%) women [20]. Despite significant progress in recent decades, women are still underrepresented as authors of research articles in medical journals [21, 22], especially as first and senior authors [23, 24]. For example, the proportion of women first authors in six prominent international medical journals increased from 6% in 1970 to 29% in 2004, and the proportion of women senior authors increased during the same period from 4% to 19% [23]. In radiology, the proportion of women first authors increased from 8% in 1978 to 32% in 2013 [25]. The proportion of women first authors in high impact general medical journals increased from 27% in 1994 to 37% in 2014, but it has recently plateaued and even declined in some journals [26].

Finally, women may be disadvantaged as the beneficiaries of research in terms of its health, societal and economic impacts [27–29]. There is evidence to suggest that research that does not account for gender differences can result in inaccurate conclusions about how women respond to disease and this, in turn, will influence the effectiveness of treatment choices [30, 31]. For example, ‘Yentl syndrome’ describes sex bias in the management of coronary heart disease due to the fact that medical research had predominantly studied symptoms of heart attacks in men [32]. Historically, those women who presented with symptoms of heart attack similar to those in men received the same diagnostic and treatment procedures as men, but those whose symptoms presented differently were not properly treated and may have died unnecessarily. For such reasons, the League of European Research Universities has recently stressed that, without including gender analysis in research, the impact of science may not be equally beneficial for both men and women [33].

In what follows, we elaborate on the path dependent nature of gender bias in science and why it is difficult, but not impossible to address. Second, we outline key characteristics of research impact assessment; we argue that by investigating gender bias, research impact assessment can become a force for good in moving science policy and practice towards gender equity. Third, we offer a set of recommendations about how research funders, research institutions and research evaluators can include and strengthen analysis of gender equity in research impact assessment. We conclude by issuing a global call for action.

Path dependency of gender bias

Historical gender biases have generated a form of path dependency such that the research system, including research impacts, is, to some extent, male oriented. In most countries, women began entering higher education institutions in substantial numbers only from the beginning of the 20th century. However, over the last century, remarkable social progress has been made, across much of the globe, towards gender equity. Proponents of the so-called ‘pipeline’ argument believed that once numbers of women entering universities reached sufficient numbers and they were not discriminated against for admission into the pipeline, gender bias in the representation of women in science would gradually disappear [34–38]. Indeed, in many countries where there are strict non-discrimination laws, women have reached 40% or more in admissions to medicine and other university health science degrees – a figure that is sometimes used as a threshold for gender balance [39]. Nevertheless, gender bias in health research still persists because the path dependent nature of science makes it difficult to change the status quo.

The notion of path dependency, which originates in economics and political science, suggests that our current actions depend on existing knowledge and past decisions, and therefore strong conjunctural forces are required to move policy and practice away from the established path [40]. Research on path dependency in political science suggests that such forces are associated with rational values and a strong centralised authority such as governments wishing to change current policy and practice [40, 41]. Path dependent gender bias in research is hard to address because it is often institutionalised as policies, practices, beliefs and written or unwritten rules of behaviour that structure modern science and society.

The path dependent tendency of science is aptly illustrated by Isaac Newton’s metaphor of ‘standing on the shoulders of giants’. Historically, these giants of science have been overwhelmingly male [42]. The ‘Matthew Matilda effect’ highlights the tendency for women’s work to be systematically omitted in the history of scientific achievements or even to be misattributed to men [43]. This is part of a wider historical tendency within a range of societies (regardless of culture, religion or political organisation) for an unequal distribution of scientific resources and power between men and women. Both the current body of scientific knowledge and current scientific practices are consequently shaped predominantly by male perceptions and norms, which are often influenced by both conscious and unconscious gender bias.

As far as scientific knowledge is concerned, there has been a propensity not to take into consideration possible sex and gender differences in the research design and analysis of clinical trials [44, 45]. However, given that both the occurrence and outcomes of a range of medical conditions differ for men and women, a large body of clinical knowledge which does not take relevant sex and gender differences into account may be flawed [44]. For example, 50 years ago, few studies included women in prospective cohort studies of all-cause or coronary heart disease mortality [46]. Although a growing number of peer-reviewed health research journals now have editorial policies requiring sex- or gender-specific reporting of results [29], there remain many journals without such policies [47].

Scientific practices and structures can also be biased against women. Male and female leaders may have different leadership traits: women are more likely to focus on collaboration and may have less interest, and fare less well, in hierarchical organisation [48, 49]. However, appointment criteria for scientific leadership are often based on classical ‘male-gendered’ traits [50–53]. Academic promotion and tenure criteria have also traditionally been based on a male-gendered career trajectory which collides with the biological clock of women wishing to raise families [54], although much progress has been made in some countries to protect against bias towards those taking career breaks. Implicit bias affects the advancement and promotion of women within the research workforce [55], while recruitment materials may be interpreted through a gender bias lens, with recommendation letters for men being read as stronger than those for women with equal qualifications [56]. Likewise, women candidates for tenured research positions are sometimes put at a disadvantage due to ‘backdoor hiring’ practices and mobilisations of informal, potentially gendered, network ties, in academic recruitment and selection [57, 58].

Nevertheless, there is an opportunity to move science policy and practice towards greater gender equity, because scientific enterprise is founded on rational values. In essence, scientific enterprise strives to produce objective knowledge without bias, with a view to bringing societal and economic benefits to humanity. Greater gender equity helps scientific enterprise achieve objective knowledge regardless of the ways objectivity is conceived of by different epistemological communities. The traditional view is that objective knowledge exists and can be achieved by removing different biases, including the gender bias. This view is questioned by feminist scholars who contend that all knowledge is gender-biased in one way or another and thus objectivity can be achieved by increasing the pluralism of perspectives to balance biases and by critically evaluating biases in both knowledge and methods of enquiry [59–62].

Although the value of gender equity for increasing the objectivity of science is acknowledged by different epistemological communities, the scope for centralised policy action to maximise research impact through the gender equity pathway within scientific enterprise is limited. This is partly because many research funders and institutions are decentralised (and cross multiple sectors including private, public and third-sector), and partly because changes in policy may reduce but not eliminate gender bias. Whilst national and local policies may be in place, the reality of research enterprise is that decision-making authority is devolved to the level of departments, centres and, sometimes, individual principal investigators. Moreover, scientific enterprise has (encouragingly) become globalised (there is increased mobility of researchers, and many research funders and institutions seek to become globally competitive in the knowledge economy). It follows that gender equity in research requires action at a global level to be maximally effective (Fig. 1).

Fig. 1View larger versionFig. 1. The gender equity pathway to maximise research impact. Shown are the forces that influence the key stakeholders in promoting gender equity to maximise different possible types of research impact

Research impact assessment

Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. Unlike the primary orientation of basic science, which is concerned with the advancement of knowledge for its own sake, research impact assessment (a form of research on research) is predominantly oriented towards applied research objectives of influencing policy and practice. Research impact assessment has developed its own methods of analysis, drawing on other disciplines and fields of knowledge [63–68]. Its combination of applied research objectives and scientific rigour should allow research impact assessment to become a force for good in moving science policy and practice towards gender equity by investigating gender bias and recommending actions for change.

The driving forces behind research impact assessment have been conceptualised into the Four “As” of Advocacy, Accountability, Analysis and Allocation [69]. Given the challenges of addressing gender bias in research, each of the Four “As” of research impact assessment are necessary to address gender bias in research (Fig. 2):

Advocacy is needed to ‘make the case’ [69] for science free from gender bias, by highlighting the scientific, societal, and economic benefits of gender equity in research.

Accountability to the public requires that funding and staffing decisions by research funders and research institutions are made fairly, and that an account of such decision-making is captured and reported transparently through appropriate metrics.

Analysis is necessary to challenge gender bias and discover policies that eliminate it, as well as conditions under which these policies could be transferred to other organisations and countries.

Allocation of research funding to ensure equitable participation of both genders in research is imperative for the legitimisation of and public support for science [38].

Fig. 2View larger versionFig. 2. The Four “As” of research impact assessment with regard to gender equity

Various methodological approaches may be used within research impact assessment to investigate and address gender bias (Table 1). For example, logic modelling and related theories of change [70] can be applied to systematically investigate gender equity at all stages of the research process, as well as effective mechanisms for ensuring the transparent and equitable distribution of resources and power. Bibliometrics and other traditional research output metrics (scientometrics) can be used to measure gender-based differences in research outputs, their academic impacts, including gender-based citation behaviour, and non-academic impacts [23, 71–74]; however, unless supplemented by additional methodologies, they will not illuminate the reasons for such differences. Potentially, the more immediate academic and non-academic impact of research can also be measured using non-traditional web-based metrics (altmetrics) [75, 76]. Surveys and narrative case studies can be especially useful in investigating these differences. Curriculum vitae and survey data can be combined to measure variations in researchers’ societal outreach and orientation [77], and text-mining algorithms can be employed to analyse developments in the scholarly attention to gender and sex issues in biomedical research [78]. Information and management science approaches can be employed to develop performance management systems (such as the recently described ‘balanced scorecard’ [79]) for assessing and monitoring gender equity both in the awarding of grants and selection of research topics. Finally, economic modelling and cost-benefit analysis could be employed to examine how gender equity can maximise economic returns on investment in research.

See full tableTable 1. Selection of methods in research impact assessment and how they may be used to investigate and address gender bias

Whichever methodological approaches are employed, active engagement with users of impact assessment is needed, in turn, to disseminate findings and influence policy and practice. Communication of these findings is likely to be strengthened by the use of infographics, diagrams, charts and other visual tools to help clearly convey qualitative and quantitative data and trends. For example, recent analysis of the United Kingdom’s Research Excellence Framework 2014 impact case studies has used innovative infographics, alluvial and chord diagrams, word clouds, heat maps and impact wheels, synthesising complex data to reveal where research has had a societal impact [80]. With increasing use of the web and social media by researchers and research users, impact assessment results can be rapidly communicated through a variety of media, including research blogs, social networks and web feeds. Gender-sensitive visualisation and dissemination can further enhance engagement with users of impact assessment.

Recommendations

For research impact assessment to become a force for good in moving science policy and practice towards gender equity, we propose that gender be routinely included in research impact assessment. Based on our knowledge and experience of the field, we – a group of scholars and practitioners representing research leaders from Africa, America, Asia and Europe – offer recommendations to research funders, research institutions and research evaluators (i.e. those who conduct research impact assessment) on how to include and strengthen analysis of gender equity in research impact assessment. Our recommendations are outlined below and summarised in Table 2.

See full tableTable 2. Recommendations to include and strengthen analysis of gender equity in research impact assessment

Research funders

Internationally, research funders have long conducted research impact assessment in order to demonstrate accountability to taxpayers and philanthropists as well as to inform their resource allocation practices. In so doing, many funders already have established systems for collecting basic gender-based information on the research they fund. Such information is often required to demonstrate compliance with anti-discrimination legislation. However, it is not routinely used for research impact assessment. We recommend that research funders conduct retrospective research impact assessment using gender-based information already available. We suggest this should inform the development and implementation of gender equity policies, and lead to more gender-sensitive prospective data collection. This would help identify research programmes and practices that are most reflective of gender equity, areas which may need may need further exploration or development, and policies to support fairer application processes. For example, the Australian Research Council has recently introduced a ‘gender equity action plan’, including policies to report gender disaggregated data and monitor the gender outcomes of selection rounds [81].

It has long been recognised that research results may not equally apply to men and women if scientific evidence fails to consider relevant sex and gender issues. However, developing formal policies and approaches by research funders to encourage researchers to include considerations of sex and gender in their research designs and analyses is a relatively recent phenomenon [82]. For example, since 2009, the Canadian Institutes of Health Research requires that all grant applicants consider whether their research designs include sex and gender when appropriate [83]. In 2014, the United States National Institutes of Health announced new supplemental awards to explore the effects of sex and gender in preclinical and clinical studies [84]. The European Commission has integrated gender- and sex-based analysis into Horizon 2020 – its biggest research and innovation programme for 2014–2020 [85]. Such policies encourage researchers to study both sexes to aid scientific innovation and better health for all citizens. We recommend that research funders adopt policies to ensure that researchers address relevant sex and gender issues in their research designs and analyses.

There are also research funders that now require research institutions applying for research funding to demonstrate how they support gender equality in research careers. This is mainly achieved through various award schemes designed to promote structural change in research institutions [86]. For example, the United Kingdom National Institute for Health Research made designation and funding of its Biomedical Research Centres conditional upon the achievement by academic applicants of at least the Silver Award of the Athena Project and the Scientific Women’s Academic Network (SWAN) Charter for Women in Science, which recognises work undertaken by research institutions to advance women’s careers in science, technology, engineering, mathematics and medicine (STEMM) [79]. Another example of integrating gender equity into research funding criteria is the report card developed by the New York Stem Cell Foundation to assess institutions in grant application processes [87]. We recommend that research funders identify and apply evidence-based approaches to integrating gender equity into research funding criteria.

So far, relatively little scholarly attention has been given to the scientific, societal and economic impact of gender equity in research. Debate on gender equity in research has been primarily driven by human rights and equality imperatives and is based on evidence that is sparse and of variable methodological quality [3]. Importantly, experimental research shows gender differences in how the quality of evidence revealing gender bias is evaluated by women and men [88]. There is a need to systematise the current evidence to map more precisely the state of knowledge, ignorance and uncertainty in the field, and identify gaps in the evidence base where new research is needed. Such a scoping exercise is likely to highlight the need for further comparative effectiveness research to determine the most effective policy interventions, as well as the conditions under which particular interventions can be effective in different settings. Additionally, fields beyond medicine (such as higher education studies and the sociology of science) may provide important theoretical perspectives that will help guide the design of empirical studies. We recommend that, where appropriate, research funders support theoretical and applied research on the scientific, societal and economic impact of gender equity in research.

Research institutions

Increasingly, research institutions realise that “winning the talent war for women” [89] has great potential to contribute to their international competitiveness and future growth. However, research institutions do not always have information systems that allow disaggregating key research performance data by gender. Moreover, progress towards gender equity and the enhanced ability to compete for the best students, faculty and staff of both genders are not routinely considered as key performance indicators by universities themselves and as a basis for competition with their peers globally, e.g. in university rankings and league tables. We recommend that universities and other research institutions establish systems for collecting and analysing gender-based information on the research process.

Gender equity in research institutions tends to vary by level of seniority, department, and discipline. Women tend to be better represented at entry and junior levels and underrepresented at senior levels, especially in the STEMM sciences and clinical trials. If gender analysis is conducted on too coarse a unit of aggregation, different biases can inadvertently cancel each other out (e.g. women in nursing, men in surgery). We recommend that research institutions work towards improving and refining the quality of gender-based data, e.g. by ensuring that it is possible to disaggregate gender-based information vertically by the level of seniority and horizontally by department and discipline.

Research institutions have a unique role to play in the inclusion of gender in research impact assessment because they educate and employ today’s and tomorrow’s research evaluators. Research institutions can create awareness amongst their students and staff about the importance of gender analysis and provide them with the necessary knowledge and skills to analyse and report gender-specific and gender-comparative results. For example, they can include elements of gender analysis and social science perspectives on gender biases in knowledge and methods of enquiry in the teaching curricula of the relevant disciplines; organise multi-disciplinary workshops and seminars to develop the competencies of the students and staff interested in research impact assessment; implement comprehensive diversity and inclusion plans [90]; consider targets and quotas for women in leadership [91]; actively develop expert knowledge leadership in gender-sensitive research impact [92]; and even evaluate and address the gender pay gap [93, 94]. We recommend that research institutions train and support interested students and staff to include gender in research impact assessment.

Most research institutions across the globe now have in place programmes and initiatives to support gender equity, either as part of their own efforts or as a response to the incentives provided by research funders. For example, most research-intensive universities in the United Kingdom participate in the Athena SWAN Charter for Women in Science and its national and regional equality networks [95]. This requires extensive data collection and analysis, discussions in working groups and committees, and preparation and implementation of action plans. Female faculty bear much of the burden for preparing Athena SWAN award applications [87], which may even be to the detriment of their core scientific activity and thus productivity and promotion. Therefore, it is important to ensure that gender equity activity undertaken by faculty is recognised within their own institutions and scientific fields. We recommend that research institutions include gender equity activity as a criterion for performance management and annual appraisal and promotion.

Research evaluators

The science of research impact assessment has already developed a number of validated evidence-based approaches that can be used for gender analysis. Moreover, there is an emerging evidence base of successful research impact assessment studies focussing specifically on gender [96, 97]. By using similar tools in new studies, research evaluators can test their robustness and extend the evidence base in ways that allow synthesis across studies. Therefore, we recommend that research evaluators engage with the research impact assessment literature with a view to identifying and applying evidence-based tools and approaches to gender-sensitive research impact assessment. However, when new kinds of research questions are being asked about gender, it may be appropriate to develop new tools and approaches.

Gender analysis can reveal important differences between men and women in research production and impact that can inform policies to optimise performance of the scientific enterprise. Whereas male scientists may be more productive during the early stages of their careers, the productivity rates of female scientists may equal or surpass those of men later in their careers [98, 99]. Whereas male physician-scientists may be more interested in basic science with long-term impacts, female physician-scientists may be more interested in clinical work with more immediate patient benefits [100]. Likewise, there may be important gender differences in co-authorship behaviour, interdisciplinary collaboration and societal impact [99]. Whereas women leaders may be more collaborative, more nurturing, and share power and uncertainty more, male leaders may be more hierarchical; and whereas male researchers may be geared towards scientific rewards and recognition, female researchers may have a greater societal orientation and output [77]. However, on the basis of the evidence in relation to the quality of women’s academic outputs and citations, “there is no evidence that women do less important work than men” [52]. Thus, where gender differences exist, this not only requires further exploration, but also highlights the need for the use of appropriate metrics that do not lead to a systematic bias in assessing the research production and impact of either men or women. We recommend that research evaluators investigate gender differences in research production and impact, especially in terms of productivity, collaboration, impact time lags and types of impact, and carefully select appropriate metrics to avoid bias.

It is important to ensure that the act of research impact assessment itself does not become affected by gender bias. For example, there may be a gender bias in the selection of methods and models of inquiry. Whereas quantitative methods and a positivist natural science model of inquiry have been traditionally associated with male researchers, qualitative methods and an interpretivist social science model of inquiry have been traditionally associated with female researchers [101]. Moreover, there is a disagreement between different epistemological communities as to whether the objectivity of knowledge can be achieved by removing gender bias or by increasing the pluralism of perspectives to balance different gender biases [60]. Regardless of the ways objectivity is conceived of by different epistemological communities, increasing gender equity in research teams and the pluralism of methods can increase the objectivity of knowledge. We recommend that research evaluators increase the objectivity of research impact assessment by conducting evaluations in gender-balanced teams using an appropriate mix of methods.

Research impact assessment is a rapidly growing field of knowledge spanning disciplinary, organisational and geographical boundaries. Research evaluators can stay abreast of new knowledge and strategic developments in the field through collaboration and knowledge sharing. These can also help identify solutions to common problems and provide feedback without wasting time and resources. Moreover, participation in boundary-spanning networks and communities of practice can diffuse innovative ideas and unleash collective creativity to discover new approaches for the greater good. In doing so it is important that networks and communities of practice strive to achieve gender balance among their members and leaders, and routinely address gender issues. We recommend that research evaluators collaborate and share knowledge through gender-balanced networks and communities of practice.

Research evaluators have a potentially transformative role to play in the scientific enterprise because research impact assessment is predominantly oriented towards influencing policy and practice. As research evaluators investigate research outputs and outcomes, they engage with a wide range of stakeholders and communicate their findings to decision-makers. This gives research evaluators an opportunity not only to investigate gender bias, but also to suggest a course of policy interventions to promote gender equity. The knowledge of science policy in various settings also enables research evaluators to initiate public debate and advocate for science free from gender bias. We recommend that research evaluators engage impact assessment users and the public in debate on gender bias and recommend actions for change.

Conclusions

Growing global investment in biomedical research is unlikely to result in outstanding science that benefits women and men equitably if current levels of conscious and unconscious gender bias in health research persist. Gender bias is difficult to eliminate, in part because of the historical late entry of women into higher education and research, and a tendency of path dependency in science. However, we argue that research impact assessment can become a force for good in moving science policy and practice towards gender equity by revealing and challenging gender bias. Success in applying research impact assessment to address gender bias will depend on sustained action by multiple stakeholders at all stages of the research process and internationally. We call on research funders, research institutions and research evaluators globally to include gender in research impact assessment in order to maximise scientific, societal and economic returns on investment in research.

Abbreviations

Athena SWAN, Athena Project and the Scientific Women’s Academic Network; STEMM, science, technology, engineering, mathematics and medicine

Acknowledgements

Early ideas for this paper were presented and discussed at the International School on Research Impact Assessment, 8–12 November 2015, Doha, organised by the Qatar National Research Fund, Alberta Innovates – Health Solutions (AIHS), Agency for Health Quality and Assessment of Catalonia (AQuAS), RAND Europe, and King’s College London. We would like to thank Mrs. Safia Al Marhoobi and Mrs. Noora Al Balushi from The Research Council (Oman) for their assistance with data collection, and Dr. Hasneen Karbalai from The University of Oxford for his valuable comments and suggestions. HLS is supported by European Commission & IGRUE FP7 award TRIGGER: Transforming Institutions by Gendering contents and Gaining Equality in Research (2014–2017) Grant Agreement n° 611034. LDE was supported by the John Fell Fund and the Vice-Chancellor’s Diversity Fund, University of Oxford. AMB, KMC and GML hold NIHR Senior Investigator Awards. The publication of this article was supported by the NIHR Oxford Biomedical Research Centre.

Funding

NIHR Oxford Biomedical Research Centre, Oxford, United Kingdom.

Availability of data and materials

Not applicable.

Authors’ contributions

PVO conceived of this call for action and led the drafting of the manuscript with substantive input from TG. All other authors critically revised the manuscript and provided important comments and suggestions. All of the authors read and approved the final version of the manuscript.

Competing interests

TG represented Medicine on the Equality and Diversity Panel for the 2014 Research Excellence Framework (REF); in that role, she was involved in assessing universities’ procedures for assuring equality (by gender, ethnicity, disability and sexual orientation) in their decisions on which individuals to submit to the REF. RS is Advocate for Diversity and Pro Vice-Chancellor, University of Oxford. RG has grant funding from the Doris Duke Charitable Foundation, the National Institutes of Health, the Michigan Radiation Oncology Quality Consortium, and the Greenwall Foundation for unrelated work. All other authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Disclaimer

The views expressed are those of the authors and not necessarily of the authors’ respective organisations, funders or sponsors.

Articles from Health Research Policy and Systems are provided here courtesy of BioMed Central

PMC Copyright Notice

The articles available from the PMC site are protected by copyright, even though access is free. Copyright is held by the respective authors or publishers who provide these articles to PMC. Users of PMC are responsible for complying with the terms and conditions defined by the copyright holder.

Users should assume that standard copyright protection applies to articles in PMC, unless an article contains an explicit license statement that gives a user additional reuse or redistribution rights. PMC does not allow automated/bulk downloading of articles that have standard copyright protection.

See the copyright notice on the PMC site, https://www.ncbi.nlm.nih.gov/pmc/about/copyright/, for further details and specific exceptions.

References

Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al., authors. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. [Cross Ref] [PubMed]

Klinge I, author. Bringing gender expertise to biomedical and health-related research. Gend Med. 2007;4(Suppl 2):S59–63. [Cross Ref] [PubMed]

Edmunds LD, Ovseiko PV, Shepperd S, Greenhalgh T, Frith P, Roberts NW, et al. Why do women choose or reject careers in academic medicine? A narrative review of empirical evidence. Lancet. 2016;pii: S0140-6736(15)01091-0. doi:10.1016/S0140-6736(15)01091-0. Ahead of print.

Pololi LH, Jones SJ, authors. Women faculty: an analysis of their experiences in academic medicine and their coping strategies. Gend Med. 2010;7(5):438–50. [Cross Ref] [PubMed]

Pololi LH, Civian JT, Brennan RT, Dottolo AL, Krupat E, authors. Experiencing the culture of academic medicine: gender matters, a national study. J Gen Intern Med. 2013;28(2):201–7. [Cross Ref] [PubMed]

Beery AK, Zucker I, authors. Sex bias in neuroscience and biomedical research. Neurosci Biobehav Rev. 2011;35(3):565–72. [Cross Ref] [PubMed]

Curno MJ, Rossi S, Hodges-Mameletzis I, Johnston R, Price MA, Heidari S, authors. A systematic review of the inclusion (or exclusion) of women in HIV research: from clinical studies of antiretrovirals and vaccines to cure strategies. J Acquir Immune Defic Syndr. 2016;71(2):181–8. [Cross Ref] [PubMed]

Jagsi R, Motomura AR, Amarnath S, Jankovic A, Sheets N, Ubel PA, authors. Under-representation of women in high-impact published clinical cancer research. Cancer. 2009;115(14):3293–301. [Cross Ref] [PubMed]

Grant J, Burden S, Breen G, authors. No evidence of sexism in peer review. Nature. 1997;390(6659):438 [Cross Ref] [PubMed]

Head MG, Fitchett JR, Cooke MK, Wurie FB, Atun R, authors. Differences in research funding for women scientists: a systematic comparison of UK investments in global infectious disease research during 1997-2010. BMJ Open. 2013;3(12):e003362 [Cross Ref] [PubMed]

Wenneras C, Wold A, authors. Nepotism and sexism in peer-review. Nature. 1997;387(6631):341–3. [Cross Ref] [PubMed]

Bornmann L, Mutz R, Daniel HD, authors. Gender differences in grant peer review: a meta-analysis. J Informetr. 2007;1(3):226–38. [Cross Ref]

Marsh HW, Bornmann L, Mutz R, Daniel H-D, O’Mara A, authors. Gender effects in the peer reviews of grant proposals: a comprehensive meta-analysis comparing traditional and multilevel approaches. Rev Educ Res. 2009;79(3):1290–326. [Cross Ref]

Turner S, Davidson P, Stanton L, Cawdeary V, authors. Features of successful bids for funding of applied health research: a cohort study. Health Res Policy Syst. 2014;12:54 [Cross Ref] [PubMed]

Sege R, Nykiel-Bub L, Selk S, authors. Sex differences in institutional support for junior biomedical researchers. JAMA. 2015;314(11):1175–7. [Cross Ref] [PubMed]

Jagsi R, Motomura AR, Griffith KA, Rangarajan S, Ubel PA, authors. Sex differences in attainment of independent funding by career development awardees. Ann Intern Med. 2009;151(11):804–11. [Cross Ref] [PubMed]

Ley TJ, Hamilton BH, authors. Sociology. The gender gap in NIH grant applications. Science. 2008;322(5907):1472–4. [Cross Ref] [PubMed]

van der Lee R, Ellemers N, authors. Gender contributes to personal research funding success in The Netherlands. Proc Natl Acad Sci U S A. 2015;112(40):12349–53. [Cross Ref] [PubMed]

Vydareny KH, Waldrop SM, Jackson VP, Manaster BJ, Nazarian GK, Reich CA, et al., authors. Career advancement of men and women in academic radiology: is the playing field level? Acad Radiol. 2000;7(7):493–501. [Cross Ref] [PubMed]

Noblelprize.org. All Nobel Prizes in Physiology or Medicine. http://www.nobelprize.org/nobel_prizes/medicine/laureates/. Accessed 1 April 2016.

Bhattacharyya N, Shapiro NL, authors. Increased female authorship in otolaryngology over the past three decades. Laryngoscope. 2000;110(3 Pt 1):358–61. [Cross Ref] [PubMed]

Amering M, Schrank B, Sibitz I, authors. The gender gap in high-impact psychiatry journals. Acad Med. 2011;86(8):946–52. [Cross Ref] [PubMed]

Jagsi R, Guancial EA, Worobey CC, Henault LE, Chang Y, Starr R, et al., authors. The “gender gap” in authorship of academic medical literature--a 35-year perspective. N Engl J Med. 2006;355(3):281–7. [Cross Ref] [PubMed]

Sidhu R, Rajashekhar P, Lavin VL, Parry J, Attwood J, Holdcroft A, et al., authors. The gender imbalance in academic medicine: a study of female authorship in the United Kingdom. J R Soc Med. 2009;102(8):337–42. [Cross Ref] [PubMed]

Piper CL, Scheel JR, Lee CI, Forman HP, authors. Gender trends in radiology authorship: a 35-year analysis. AJR Am J Roentgenol. 2016;206(1):3–7. [Cross Ref] [PubMed]

Filardo G, da Graca B, Sass DM, Pollock BD, Smith EB, Martinez MA, authors. Trends and comparison of female first authorship in high impact medical journals: observational study (1994-2014). BMJ. 2016;352:i847 [Cross Ref] [PubMed]

Sen G, Östlin P, George A. Unequal, unfair, ineffective and inefficient gender inequity in health: why it exists and how we can change it. Final Report to the WHO Commission on Social Determinants of Health. http://www.who.int/social_determinants/resources/csdh_media/wgekn_final_report_07.pdf. Accessed 1 April 2016.

Kuhlmann E, Annandale E, authors; Kuhlmann E, Blank RH, Bourgeault IL, Wendt C, editors. Gender and healthcare policy. The Palgrave international handbook of healthcare policy and governance. 2015. Basingstoke: Palgrave Macmillan; p. 578–96

Schiebinger L, Klinge I, Sánchez de Madariaga I, Paik HY, Schraudner M, Stefanick M. Gendered innovations in science, health & medicine, engineering, and environment. http://genderedinnovations.stanford.edu/. Accessed 1 April 2016.

Bartlett C, Doyal L, Ebrahim S, Davey P, Bachmann M, Egger M, et al., authors. The causes and effects of socio-demographic exclusions from clinical trials. Health Technol Assess. 2005;9(38):iii–iv. [Cross Ref] [PubMed]

Johnson PA, Fitzgerald T, Salganicoff A, Wood SF, Goldstein JM. Sex-specific medical research: why women’s health can’t wait. A Report of the Mary Horrigan Connors Center for Women’s Health & Gender Biology at Brigham and Women’s Hospital. http://www.brighamandwomens.org/Departments_and_Services/womenshealth/ConnorsCenter/Policy/ConnorsReportFINAL.pdf. Accessed 1 April 2016.

Healy B, author. The Yentl syndrome. N Engl J Med. 1991;325(4):274–6. [Cross Ref] [PubMed]

Buitendijk S, Maes K. Gendered research and innovation: integrating sex and gender analysis into the research process. http://www.leru.org/files/publications/LERU_AP18_Gendered_research_and_innovation_final.pdf. Accessed 1 April 2016.

Etzkowitz H, Kemelgor C, Uzzi B, authors. Athena unbound: the advancement of women in science and technology. 2000. Cambridge: Cambridge University Press;

Ley TJ, Rosenberg LE, authors. The physician-scientist career pipeline in 2005: build it, and they will come. JAMA. 2005;294(11):1343–51. [Cross Ref] [PubMed]

Puljak L, Kojundzic SL, Sapunar D, authors. Gender and academic medicine: a good pipeline of women graduates is not advancing. Teach Learn Med. 2008;20(3):273–8. [Cross Ref] [PubMed]

Sexton KW, Hocking KM, Wise E, Osgood MJ, Cheung-Flynn J, Komalavilas P, et al., authors. Women in academic surgery: the pipeline is busted. J Surg Educ. 2012;69(1):84–90. [Cross Ref] [PubMed]

Etzkowitz H, Kemelgor C, Neuschatz M, Uzzi B, Alonzo J, authors. The paradox of critical mass for women in science. Science. 1994;266(5182):51–4. [Cross Ref] [PubMed]

Commission of the European Communities, author. Women in science: the gender dimension as a leverage for reforming science. 2001. Brussels: Commission of the European Communities;

Wilsford D, author. Path dependency, or why history makes it difficult but not impossible to reform health care systems in a big way. J Public Policy. 1994;14(3):251–83. [Cross Ref]

Alexander AC, Welzel C, authors. Empowering women: four theories tested on four different aspects of gender equality. 2007. Chicago: Annual Meeting of Midwest Political Science Association;

Schiebinger L, editor. Women and gender in science and technology. 2014. London: Routledge;

Rossiter MW, author. The Matthew Matilda effect in science. Soc Stud Sci. 1993;23(2):325–41. [Cross Ref]

Holdcroft A, author. Gender bias in research: how does it affect evidence based medicine? J R Soc Med. 2007;100(1):2–3. [Cross Ref] [PubMed]

Lagro-Janssen T, author; Kuhlmann E, Annandale E, editors. Sex, gender and health: developments in medical research. The Palgrave handbook of gender and healthcare. 2012. Basingstoke: Palgrave Macmillan; p. 439–54

Barrett-Connor E, author. Gender differences and disparities in all-cause and coronary heart disease mortality: epidemiological aspects. Best Pract Res Clin Endocrinol Metab. 2013;27(4):481–500. [Cross Ref] [PubMed]

Gahagan J, Gray K, Whynacht A, authors. Sex and gender matter in health research: addressing health inequities in health research reporting. Int J Equity Health. 2015;14:12 [Cross Ref] [PubMed]

Eagly AH, Johannesen-Schmidt MC, van Engen ML, authors. Transformational, transactional, and laissez-faire leadership styles: a meta-analysis comparing women and men. Psychol Bull. 2003;129(4):569–91. [Cross Ref] [PubMed]

Smith-Doerr L, author. Women’s work: gender equality vs. hierarchy in the life sciences. 2004. Boulder: Lynne Rienner;

Marchant A, Bhattacharya A, Carnes M, authors. Can the language of tenure criteria influence women’s academic advancement? J Womens Health (Larchmt). 2007;16(7):998–1003. [Cross Ref] [PubMed]

Van den Brink M, author. Scouting for talent: appointment practices of women professors in academic medicine. Soc Sci Med. 2011;72(12):2033–40. [Cross Ref] [PubMed]

de Melo-Martin I, author. Patenting and the gender gap: should women be encouraged to patent more? Sci Eng Ethics. 2013;19(2):491–504. [Cross Ref] [PubMed]

Lawton Smith H, Etzkowitz H, Meschitti V, Poulovassilis A, authors; Henry C, Nelson T, Lewis K, editors. Female academic entrepreneurship and commercialisation: reviewing the evidence and identifying the challenges. The Routledge companion to global female entrepreneurship. 2017. London: Routledge;

Fox G, Schwartz A, Hart KM, authors. Work-family balance and academic advancement in medical schools. Acad Psychiatry. 2006;30(3):227–34. [Cross Ref] [PubMed]

Committee on Maximizing the Potential of Women in Academic Science and Engineering (U.S.), author. Beyond bias and barriers: fulfilling the potential of women in academic science and engineering. 2007. Washington, D.C: National Academies Press;

Schmader T, Whitehead J, Wysocki VH, authors. A linguistic comparison of letters of recommendation for male and female chemistry and biochemistry job applicants. Sex Roles. 2007;57(7-8):509–14. [Cross Ref] [PubMed]

Nielsen MW. Limits to meritocracy? Gender in academic recruitment and promotion processes. Sci Publ Policy. 2015:10.1093/scipol/scv1052.

Husu L, author. Gender discrimination in the promised land of gender equality. High Educ Eur. 2000;25(2):221–8. [Cross Ref]

Harding SG, author. The science question in feminism. 1986. Ithaca: Cornell University Press;

Longino HE, author. Science as social knowledge: values and objectivity in scientific inquiry. 1990. Princeton: Princeton University Press;

Harding SG, author. Whose science? Whose knowledge? Thinking from women’s lives. 1991. Ithaca: Cornell University Press;

Nelson LH, Nelson J, editors. Feminism, science and the philosophy of science. 1996. Dordrecht: Kluwer Academic;

Kostoff RN, author. Research impact assessment. Principles and applications to proposed, ongoing, and completed projects. Invest Radiol. 1994;29(9):864–9. [Cross Ref] [PubMed]

Fealing KH, Lane JI, Marburger JHI, Shipp SS, editors. The science of science policy: a handbook. 2011. Stanford: Stanford University Press;

Banzi R, Moja L, Pistotti V, Facchini A, Liberati A, authors. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26 [Cross Ref] [PubMed]

Ovseiko PV, Oancea A, Buchan AM, authors. Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators. BMC Health Serv Res. 2012;12:478 [Cross Ref] [PubMed]

Guthrie S, Wamae W, Diepeveen S, Wooding S, Grant J. Measuring research: a guide to research evaluation frameworks and tools. http://www.rand.org/pubs/monographs/MG1217.html. Accessed 1 April 2016.

Milat AJ, Bauman AE, Redman S, authors. A narrative review of research impact assessment models and methods. Health Res Policy Syst. 2015;13:18 [Cross Ref] [PubMed]

Morgan Jones M, Grant J, authors; Dean A, Wykes M, Stevens H, editors. Making the grade: methodologies for assessing and evidencing research impact. 7 essays on impact. DESCRIBE project report for JISC. 2013. Exeter: University of Exeter; p. 25–43

Coryn CLS, Noakes LA, Westine CD, Schröter DC, authors. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2011;32(2):199–226. [Cross Ref]

Pohlhaus JR, Jiang H, Wagner RM, Schaffer WT, Pinn VW, authors. Sex differences in application, success, and funding rates for NIH extramural programs. Acad Med. 2011;86(6):759–67. [Cross Ref] [PubMed]

Eloy JA, Svider PF, Kovalerchik O, Baredes S, Kalyoussef E, Chandrasekhar SS, authors. Gender differences in successful NIH grant funding in otolaryngology. Otolaryngol Head Neck Surg. 2013;149(1):77–83. [Cross Ref] [PubMed]

Lariviere V, Ni C, Gingras Y, Cronin B, Sugimoto CR, authors. Bibliometrics: global gender disparities in science. Nature. 2013;504(7479):211–3. [Cross Ref] [PubMed]

McMillan GS, author. Gender differences in patenting activity: An examination of the US biotechnology industry. Scientometrics. 2009;80(3):683–91. [Cross Ref]

Altmetric LLP. About altmetrics: an introduction to altmetrics and how they can be used. https://www.altmetric.com/about-altmetrics/. Accessed 1 April 2016.

Bar-Ilan J, van der Weiden I, authors. Altmetric gender bias? An exploratory study. Int J Comput Sci. 2015;4(1):16–22

Van der Weijden ICM, Zahedi Z, Must Ü, Meijer I. Gender differences in societal orientation and output of individual scientists. In: Proceedings of the 19th International Conference on Science and Technology Indicators; Leiden: Universiteit Leiden; 3–5 September 2014. p. 680–6.

Oertelt-Prigione S, Parol R, Krohn S, Preissner R, Regitz-Zagrosek V, authors. Analysis of sex and gender-specific research reveals a common increase in publications and marked differences between disciplines. BMC Med. 2010;8:70 [Cross Ref] [PubMed]

Ovseiko PV, Edmunds LD, Pololi LH, Greenhalgh T, Kiparoglou V, Henderson LR, et al., authors. Markers of achievement for assessing and monitoring gender equity in translational research organisations: a rationale and study protocol. BMJ Open. 2016;6(1):e009022 [Cross Ref] [PubMed]

King’s College London, Digital Science. The nature, scale and beneficiaries of research impact: an initial analysis of Research Excellence Framework (REF) 2014 impact case studies. http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/Analysis,of,REF,impact/Analysis_of_REF_impact.pdf. Accessed 1 April 2016.

Australian Research Council. Gender equality in research: ARC gender equality action plan 2015–16. http://www.arc.gov.au/sites/default/files/filedepot/Public/Policy%20&%20Strategy/Gender%20Equality/ARC_Gender_Equality_Action%20Plan.pdf. Accessed 1 April 2016.

Johnson J, Sharman Z, Vissandjee B, Stewart DE, authors. Does a change in health research funding policy related to the integration of sex and gender have an impact? PLoS One. 2014;9(6):e99900 [Cross Ref] [PubMed]

Canadian Institutes of Health Research. Sex, gender and health research guide: a tool for CIHR applicants. http://www.cihr-irsc.gc.ca/e/32019.html. Accessed 1 April 2016.

National Institutes of Health. New supplemental awards apply sex and gender lens to NIH-funded research. http://www.nih.gov/news-events/news-releases/new-supplemental-awards-apply-sex-gender-lens-nih-funded-research. Accessed 1 April 2016.

European Commission. Gender equality in Horizon 2020. Version 1.0. http://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/hi/gender/h2020-hi-guide-gender_en.pdf. Accessed 1 April 2016

Equality Challenge Unit, author. GENDER-NET analysis report: award schemes, gender equality and structural change. 2015. London: Equality Challenge Unit;

Smith KA, Arlotta P, authors. Watt FM; Initiative on Women in Science and Engineering Working Group, Solomon SL. Seven actionable strategies for advancing women in science, engineering, and medicine. Cell Stem Cell. 2015;16(3):221–4. [Cross Ref] [PubMed]

Handley IM, Brown ER, Moss-Racusin CA, Smith JL, authors. Quality of evidence revealing subtle gender biases in science is in the eye of the beholder. Proc Natl Acad Sci U S A. 2015;112(43):13201–6. [Cross Ref] [PubMed]

McCracken DM, author. Winning the talent war for women. Sometimes it takes a revolution. Harv Bus Rev. 2000;78(6):159–60. [PubMed]

Davidson S, author. CSIRO diversity and inclusion plan 2012–15. 2013. Brisbane: CSIRO;

Whelan J, Wood R. Targets and quotas for women in leadership: a global review of policy, practice, and psychological research. https://cel.edu.au/our-research/targets-and-quotas-for-women-in-leadership. Accessed 1 April 2016.

Fischer MD, Dopson S, Fitzgerald L, Bennett C, Ferlie E, Ledger J, et al., authors. Knowledge leadership: mobilizing management research by becoming the knowledge object. Hum Relat. 2016;69(7):1563–85. [Cross Ref]

Jagsi R, Griffith KA, Stewart A, Sambuco D, DeCastro R, Ubel PA, authors. Gender differences in the salaries of physician researchers. JAMA. 2012;307(22):2410–7. [Cross Ref] [PubMed]

Wright AL, Ryan K, St Germain P, Schwindt L, Sager R, Reed KL, authors. Compensation in academic medicine: progress toward gender equity. J Gen Intern Med. 2007;22(10):1398–402. [Cross Ref] [PubMed]

Equality Challenge Unit. Athena SWAN members. http://www.ecu.ac.uk/equality-charters/athena-swan/athena-swan-members/. Accessed 1 April 2016.

European Commission. Gender impact assessment of the specific programmes of the Fifth Framework Programme – an overview. ftp://ftp.cordis.europa.eu/pub/science-society/docs/women_gender_impact_fp5_en.pdf. Accessed 1 April 2016.

European Commission. Monitoring progress towards gender equality in the 6th Framework Programme – synthesis report. https://ec.europa.eu/research/science-society/document_library/pdf_06/gender-monitoring-studies-synthesis-report_en.pdf. Accessed 1 April 2016.

Eloy JA, Svider P, Chandrasekhar SS, Husain Q, Mauro KM, Setzen M, et al., authors. Gender disparities in scholarly productivity within academic otolaryngology departments. Otolaryngol Head Neck Surg. 2013;148(2):215–22. [Cross Ref] [PubMed]

Elsevier Analytical Services. Mapping gender in the German research arena. https://www.elsevier.com/research-intelligence/resource-library/gender-2015. Accessed 1 April 2016.

Cydulka RK, D’Onofrio G, Schneider S, Emerman CL, Sullivan LM, authors. Women in academic emergency medicine. Acad Emerg Med. 2000;7(9):999–1007. [Cross Ref] [PubMed]

Oakley A, author. Gender, methodology and people’s ways of knowing: some problems with feminism and the paradigm debate in social science. Sociology. 1998;32(4):707–31. [Cross Ref]

[Back]

Fig. 1.

The gender equity pathway to maximise research impact. Shown are the forces that influence the key stakeholders in promoting gender equity to maximise different possible types of research impact

[Back]

[Back]

Fig. 2.

The Four “As” of research impact assessment with regard to gender equity

[Back]

[Back]

Table 1.

Selection of methods in research impact assessment and how they may be used to investigate and address gender bias

Methods | Applications
Logic modelling describes and graphically represents the chains of results associated with all stages of the research process; theories of change are formulated on the basis of these chains of results to explain how to achieve the desired results | To provide a framework to systematically investigate gender equity at all stages of the research process, and investigate how gender equity can help maximise desired research outputs and impacts
Scientometrics pertain to the statistical analysis of articles and citations in academic journals (bibliometrics) and other research-based outputs, e.g. patents, commercialisation | To measure gender-based differences in research outputs, their academic impacts, including gender-biased citation behaviour, and non-academic impacts
Altmetrics measure the online attention to academic articles on social media, mainstream news websites, blogs and social bookmarking websites | To measure gender-based differences in more immediate academic and non-academic impact of research
Surveys, curriculum vitae data, narrative case studies and text-mining algorithms collect quantitative and qualitative information that may not already have been captured as part of the research process | To investigate gender-based differences in perceptions of and approaches to the research process, outcomes and impacts
Information and management science approaches allow developing balanced scorecards and other performance management systems | To assess and monitor gender equity in research organisations
Economic modelling categorises and cost-benefit analysis assesses different types of benefits from research | To examine how gender equity can maximise economic returns on investment in research

[Back]

[Back]

Table 2.

Recommendations to include and strengthen analysis of gender equity in research impact assessment

Stakeholders
Recommendations

Research funders
• Conduct retrospective and prospective observational gender-based research impact assessment to inform implementation of gender equity policies
• Adopt policies to ensure that researchers address relevant sex and gender issues in their research designs and analyses
• Identify and apply evidence-based approaches to integrating gender equity into research funding criteria
• Where appropriate, support theoretical and applied research on the scientific, societal and economic impact of gender equity in research

Research institutions
• Establish information systems for collecting and analysing gender-based information on the research process
• Work towards improving and refining the quality of gender-based data
• Train and support interested students and staff to include gender in research impact assessment
• Establish gender equity activity as a criterion for performance management and annual appraisal and promotion

Research evaluators
• Engage with the research impact assessment literature with a view to identifying and applying evidence-based tools and approaches to gender-sensitive research impact assessment
• Identify, investigate and address gender differences in research production and impact, including commercialisation
• Increase the objectivity of research impact assessment by conducting evaluations in gender-balanced teams using an appropriate mix of methods
• Collaborate and share knowledge through gender-balanced networks and communities of practice
• Engage impact assessment users and the public in debate on gender bias and recommend actions for change

[Back]