Policy-relevant systematic reviews to strengthen health systems : models and mechanisms to support their production

Evidence & Policy • vol 12 • no 2 • 235–59 • © Policy Press 2016 • #EVPOL Print ISSN 1744 2648 • Online ISSN 1744 2656 • http://dx.doi.org/10.1332/174426415X14399963605641 This article is distributed under the terms of the Creative Commons Attribution 4.0 license (http:// creativecommons.org/licenses/by/4.0/) which permits adaptation, alteration, reproduction and distribution without further permission provided the original work is attributed. The derivative works do not need to be licensed on the same terms.


Background
For over 30 years systematic reviews have been hailed as a mechanism for informing decisions about health care.The growth of systematic reviews within the health sector has been well supported, albeit mainly to inform decisions about the effects of clinical practice.However, production and uptake of systematic reviews for decisions about the organisation and delivery of services has been slower and has faced distinct challenges.
Systematic reviews can be considered relevant to policy (and policy makers) when they present findings clearly for policy audiences to: illuminate policy problems; challenge or develop policy assumptions; or offer evidence about the impact or implementation of policy options; and take into account diversity of people and contexts.However, systematic review methods are more or less advanced across the full range of policy-relevant reviews.Moreover, institutional mechanisms (such as formal guidance, peer review procedures, templates for reporting and other agreed ways of preparing systematic reviews, supporting review teams or using reviews) are available mainly for systematic reviews of the effects of intervention.

research
The degree to which a systematic review is considered 'policy-relevant' is a matter of judgment often made by policy makers themselves when reading completed reviews.Seeking to make a review 'policy-relevant' rests, it has been argued, on reviewers sharing many judgments made at different stages in the review process with other people: those who make policy, those who implement policy or those who experience the consequences of policy (Rees and Oliver, 2012).A common barrier to the use of research evidence for policy is the mismatch between the speed of policy and research timescales (Oliver et al, 2014).In debating the tension between conducting comprehensive systematic reviews and completing them in time to meet policymaking deadlines, Thomas et al (2013) suggested some solutions that rely on policy makers and systematic reviewers working together closely.Yet Lavis et al (2005) raised pertinent questions about the implications of systematic reviewers working closely with managers and policy makers: the appropriate timing of engaging managers and policy makers for maximising rigour and relevance; the potential influence of managers and policy makers on the focus and breadth of the review question and eligible literature in light of the consequent methodological challenges for the review; the balance between contributing to a cumulative global stock of systematic reviews or providing context specific evidence; and appropriate presentations of evidence for policy consideration.
The purpose of the current study is to investigate the perspectives of policy makers, systematic reviewers and others who work with and between them to identify barriers and solutions to working together.This study asks: 1. How can systematic reviewers work with health systems policy makers?2. What are the institutional mechanisms required to produce policy relevant systematic reviews for health systems?

Methods
This study was approved by our Faculty Research Ethics Committee.
Interviews and document analysis elicited information about editorial pathways for health systems reviews, methodological support, practical resources, and training and support materials for review teams and policy makers.We inspected websites hosted by policy makers and systematic review organisations for references to the production of policy-relevant systematic reviews.
We sought key informants spanning the six World Health Organisation (WHO) regions which, between them, brought direct experience of different approaches to funding, supporting, producing and using policy-relevant reviews on health systems.We drew on the WHO's definition of health systems (2007) to define health systems research as research addressing the systemic interactions of organisations, people and actions conducted with the primary intent of informing the promotion, restoration or maintenance of health.These broad definitions include health aspects of (international) development policy and research.Initial interviewees were known to the research team, while others were identified by the Advisory Group on Research Synthesis for the Alliance of Health Policy and Systems Research (Langlois et al, 2015) or through snowballing.
Twenty-one key informants were invited to participate.No one refused but practical negotiations progressed too slowly for three interviews to be confirmed.We conducted 18 in-depth interviews spanning the six WHO regions (Table 1).Interviewees drew on their experience of health systems research (6), health systems policy (5), international development research (5), and development policy (2).Interviewees from Europe included staff of systematic review facilities: the Cochrane Effective Practice and Organisation of Care (EPOC) Review Group; the International Initiative for Impact Evaluation (3ie) and Campbell International Development Review Group; and the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre).Another interview was with a similar facility for rapid reviews, the Sax Institute in the Western Pacific.Interviewing was shared between the two authors to match their respective past experience of policy decision making at the local or national level.Both authors have long experience of conducting or supporting policy-relevant systematic reviews for strengthening health systems.Ten interviews were conducted face-to-face, seven by Skype and one by telephone.
Interviewees were invited to talk about their experience of conducting or supporting health systems systematic reviews, or of efforts to inform policy decisions with health systems research.Topic guides covered perceptions and adoption of policy-relevant systematic reviews, and collaborative working and institutional mechanisms for policy-relevant systematic reviews.To capture efficiently content, tone and emphasis, authors combined taking interview notes with repeated listening to, and partial transcribing of, sound recordings of their own interviews.Analysis began with the authors identifying themes from their own interviews.A constant comparative analysis, initiated by one author and checked by another, identified themes across interviews, to identify barriers and opportunities to producing policy-relevant reviews.Themes were further elucidated with published and unpublished documents signposted by interviewees or directly illustrating their contributions.
Analysis began before interviewing was completed.See Table 1 for further details.
Comparing themes from different perspectives, and seeking to understand the interactions between them, identified institutional mechanisms grouped within higherlevel themes: demand and supply of systematic reviews, and mutual understanding between the producers and users of reviews.
These themes were then framed in terms of two overlapping social worlds which shared dimensions of motivations, engagement methods, structures and procedures to support the demand and supply of reviews.This analytical framework, developed from our findings, was inspired by earlier work investigating the relationship between two other social worlds: community development and public health intervention (O' Mara-Eves et al, 2013).We sought to triangulate the data so that each dimension was viewed from different standpoints.As the framework was populated with findings, we purposely sought interviewees well placed to fill gaps in our understanding and, when necessary, returned to interviewees for additional information.The final framework presented high level institutional mechanisms that suited both demand and supply of systematic reviews.

Findings
The findings of the analysis are presented below first by describing the mismatch between two different worlds of policy and research, and then by describing the institutional mechanisms that operate at the interface of these two worlds.

The worlds of policy and research
Interviewees spoke of policy and research as two worlds, with systematic reviews having a low profile in both worlds and suiting better the world of research than the world of policy.Systematic reviews were rarely included in academic curricula (national policy maker, Upper Middle Income Country (uMIC)); nor were they familiar to policy makers (national policy maker Low Income Country (LIC)), some of whom, even if aware of them, considered developing their capacity for using primary research a more appropriate step either before or instead of systematic reviews (local policy maker uMIC; systematic review uMIC).Where few people within government were aware of them, systematic reviews were not prioritised even for clinical questions (reviewer, uMIC).Where there was institutional commitment to systematic reviews, strategies were still required to encourage policy teams to engage (international policy maker, High Income Country (HIC)).
Where policy makers saw the potential for systematic reviews as a policy-relevant tool, they found that reviews rarely focused on questions from policy makers (review facility, HIC).Conversely, databases of summaries of systematic reviews judged their quality only in methodological terms, and not in terms of what could be justifiably learnt for informing policy. 1 2 3 Policy makers have frequently asked broad, complex questions, where the pathway from intervention to final outcomes of interest is very long (HIC review facility).In such circumstances, simple conceptual frameworks, typical of systematic reviews to inform practice, were inadequate.Broad, complex questions unearthed overwhelming numbers of studies and were seen as the cause of major delays in the review process (HIC review facility; Rose and Battock, 2012).Manageable review questions which adopted narrow inclusion criteria risked producing reviews with no eligible studies (Yaffe et al, 2012) and thereby forfeited relevance for rigour.For instance, the author of a narrow review of the impact of community-based health insurance in L&MICs claimed the most important conclusion was the lack of convincing evidence (Ekman, 2004); whereas a broader review provided a summary of the strategies that have been used for expanding health insurance coverage, and thereby gave policy makers a series of options (Meng et al, 2011) (reviewer uMIC).
Policy makers typically asked questions that did not fit neatly into academic disciplines.
It's basic sciences, it's operational, and [it's] health systems … (uMIC policy maker) Nor have their demands been well matched by the development of methods for synthesising research.We encountered no fundamental argument against using systematic reviews for developing policy.
It's a kind of philosophical rationale, it would be inappropriate for me to do my job if I approached it in an irrational fashion, and part of rationality is evidence… Organisation functions like that, right?The organisation asks questions at that organisational level, what is the evidence for what you are doing?(local policy maker uMIC) However, we encountered problems in making it a reality even amongst enthusiasts.Local policy makers found systematic reviews inaccessible and insensitive to the context or to the confounding effects of packaging multiple interventions.
[In] health systems research, there's not enough standardised health systems methodology [so] that you can… contextualise health systems decisions.That's part of the problem… So the health systems field for systematic reviews, from where I am sitting, is very, very (pause) immature.(local policy maker uMIC) For example one intervention might be effective, another has almost been evaluated in a silo sort of way, but the question to us is what happens if you put it together as a package, what is the effect of one intervention on another?(local policy maker, uMIC) Taking account of context was not easy.
It is a challenge for researchers, how to interpret the transferability of findings from other countries.We tried to use SUPPORT tools [Lavis et al, 2009], but it needs a lot of judgement and knowledge of other country contexts.We need guidelines [or] tools.(uMIC reviewer) When shown a paper about the applicability of interventions and transferability of findings from other contexts (Wang et al, 2006), this interviewee considered: This tool is helpful, especially because… there is an example to help me understand… [but] I still think that the use of these criteria needs an understanding of the [foreign] countries' context.(uMIC reviewer) The time required for producing systematic reviews was commonly underestimated.One funder began with an expectation of six months but subsequently acknowledged that 12-18 months may be more realistic for many reviews (Rose and Battock, 2012).Local policy makers spoke of priorities that were too urgent to wait for systematic reviews.If evidence was required because 'certain policy decisions need to be made, not even within two weeks [but] in 20 minutes, because you've got a crisis' (policy maker uMIC), or even within 3-6 months (review facility), 'there's no way you are going to use a systematic review unless there is already one pre-dated, and it's readily available' (policy maker uMIC).
Addressing the problem of framing review questions that are both manageable and productive requires potential review users and review teams to understand each other.Yet some interviewees revealed little interest in, or sensitivity towards their counterparts in the 'other' world.

Institutional mechanisms to support policy-relevant reviews
Our analysis identified institutional mechanisms for bridging these two worlds that influenced the different motivations experienced within these worlds, the ways in which individuals and organisations engaged with each other across the divide, and the structures and procedures established on either side.This is illustrated in Figure 1, the framework inspired by community engagement research to investigate policy-engaged research here (O' Mara-Eves et al, 2013).Two overlapping triangles represent the two worlds, and institutional mechanisms for linking the diverse and complex worlds of policy and research with systematic reviews are arranged within different stages (columns).Institutional mechanisms are ordered within the columns from 'more' to 'less' policy development and then from 'less' to 'more' systematic reviewing, as one reads down the framework.The bi-directional arrows between the columns indicate that progression is not linear but iterative.For example, where structures place systematic reviewers in policy organisations (column 4) they are more closely aligned with the timescales motivating policy makers (column 2).Similarly, evidence-literate policy makers (column 5) design better guidance for commissioning systematic reviews (column 4).

Recognising diversity within the worlds of policy and research
Mismatches between policy makers' demands for evidence and the availability of health systems systematic reviews could be explained in part by distinguishing the diversity of policy interests and systematic review products.The geographical scale of policy makers' interests (local, national or international) had implications for their motivations and approaches for engaging with research, and their structures and institutional procedures for doing so, all of which are considered in sections below.Some policy makers were more able to span the boundary between systematic reviews and policy because their past or current career included academic appointments (Evidence into Action and What Works team; three uMIC policy makers).Some systematic reviewers have similarly crossed boundaries, having been employed or seconded to policy organisations in the past (Review facility, uMIC reviewer).
Interviewees spoke about a range of systematic reviews, particularly the value of systematic reviews that provided a bigger picture than tightly focused reviews typically informing clinical practice.Institutional support for Cochrane reviews of health systems has gradually evolved to encourage review teams to start with broad questions and split reviews if they become unwieldy. 43ie has developed evidence 'gap maps' to highlight areas where there is strong, weak or non-existent evidence. 5 The EPPI-Centre routinely encouraged a 'two-stage' review process for bridging the gap between evidence and policy with a descriptive map of the literature followed by a more in-depth review, the focus of which can be decided with policy makers and other stakeholders. 6  We commission reviews… all two-stage reviews, because there is not enough literature to go straight to a meta-analysis, but there is a large literature that can be synthesised.(HIC review funder) Embarking on reviews of previously uncharted literatures had implications for the choice of review design and methods.
They're called systematic because they're transparent, and because they treat every piece of research the same way -that's not the same as every review treating studies in the same way … (uMIC reviewer)

Harnessing motivation to achieve policy-relevant systematic reviews
National and international policy makers described systematic reviews playing an important role in setting policy directions.Motivation for policy use of reviews was formalised with financial levers when the operational arm of a Latin American government department was financed by a percentage of programme budgets to ensure they were evidence-based (uMIC policy maker).Without evidence to underpin programmes, costs were not covered by central funding.Unlike evidence comparing policy options for tackling enduring problems, such as vaccination or emergency contraception, evidence required to inform financing systems was lacking (uMIC policy maker).
Local policy makers' interest in the implementation of policies was satisfied when local officials listened to an academic combining the findings of systematic reviews with a local pilot study (reviewer uMIC).There were other drivers that thwarted research use, for instance where a policy maker's personal interest leant towards those interventions that could be established and operational quickly enough to earn them promotion.In contrast, individuals could be motivated within policy organisations when a strong commitment to using evidence was integrated into professional development, formally in a competency framework 7 and informally: If [staff] get personal kudos for using evidence and it becomes part of their professional integrity this can strengthen the incentive to use evidence.(Evidence into Action and What Works Team 2014) Distinct from the idea of systematic reviews being conducted for particular decisions (or personal gain), was the concept of evidence being made available internationally (2 review facilities, 1 international HIC policy maker) in the form of systematic reviews as 'public goods' 8 (review facility) or evidence-informed guidance (HIC reviewer).When not facing a specific urgent decision, the aspiration was to produce systematic reviews of high technical quality for wide use.'Protecting the brand' was a phrase applied by different systematic review programmes with such a mission.At the same time, review and policy organisations recognised that the need for evidence was often widespread so, once completed, review reports were not only made publicly available on funders' websites, review facilities websites, and through searchable databases, but also translated into concise, readable messages (Department for International Development (DFID) evidence briefs, SUPporting POlicy relevant Reviews and Trials (SUPPORT) summaries, Supporting the Use of Research Evidence (SURE) guides).

Emphasising engagement between the worlds of policy and research
Getting the 'management buy-in cycle' right for systematic reviews required researchers to have the technical capacity to address directly the needs of the policy department (local policy maker uMIC).However, the two schools of thought [policy and research] don't quite understand each other… [Locally] I think we've come a long way in bridging that gap… [through] constant engagement, constant working together, and just building those relationships… getting to know each.[uMIC) Institutional mechanisms involved clear expectations of each world being captured in the production of systematic reviews, with formal roles for policy makers and priority setting, contracts between funders and reviewers, and training and support for mutual engagement.

Relationships for shaping systematic reviews
In an ideal world, you would not be commissioning a SR unless you've already had a conversation with a practitioner and policy maker and you know that the review is coming from a need of somebody trying to change the system.(local policy maker uMIC) Policy makers needed time, guidance, tools and training to shape questions before commissioning systematic reviews (Rose and Battock, 2012;DFID response, 2013).When policy leads have lacked experience, reviewers have invested considerable time in negotiating review questions (uMIC and Lower Middle Income Country (lMIC) reviewers) face-to-face or, less successfully, by telephone (uMIC reviewer).Inexperience on both sides has led to lengthy negotiation (lMIC reviewer).For the review question and scope there was 'a tension between being too inclusive and inefficient or being too focused and missing important data' (uMIC reviewer).Experts in review methods balanced experts in the subject area and had to resist the enthusiasm from policy leads for expanding questions (review facility).The subsequent challenge for health systems reviews was 'identifying a framework to interpret what we find… a Population, Intervention, Comparison, Outcome (PICO) framework [which suits clinical practice reviews] is insufficient'.Developing the framework benefited from input from the relevant stakeholders.When they brought a clinical or public health background they often needed help in thinking in terms of systems (uMIC reviewer).
An alternative to choosing review questions one at a time was developing a list of priorities.This started either with listening to policy makers, then comparing their priorities with existing evidence (Morton et al, 2012;Cheung et al, 2011;Cumpston et al, 2012) or conducting an overview of systematic reviews then consulting partner organisations (Munabi-Babigumira et al, undated).

Emphasising research-policy engagement
The Latin American model mentioned above, which required policy makers to justify their programmes with systematic reviews, ensured sufficient funds to sustain a team were available for producing systematic reviews to meet policy needs (uMIC policy maker).In the UK, business cases prepared by policy makers for major investments were expected to use synthesised evidence products (Evidence into Action and What Works teams, 2014).Two-stage reviews provided a formal opportunity for policy input after scoping the literature and before embarking on detailed appraisal and synthesis.If each stage were to be commissioned separately, the time and cost required to carry out a full systematic review could be more accurately estimated in advance, although two-stage contracts with break points may deter potential review teams from bidding (Rose and Battock, 2012;DFID response, 2013).

Training and support for engagement
Where engagement was seen as a key element of either policy decision making or generating evidence, it was integrated into training for policy makers and researchers either on the initiative of an enthusiastic individual (lMIC policy maker) or as part of organisational policy (HIC policy maker).
On-going support for review teams included peer review of the protocol and draft report.Policy makers could be invited to comment on all aspects of the work.Alternatively, four or five… more specific questions about relevance to policy areas are probably more helpful.(HIC review facility) Some review facilities formally structured peer review comments into Obligatory and optional.Most policy comments are obligatory and [the review facility] will provide suggestions on how they can be addressed.(HIC review facility) Review facilities have also supported discussions between policy makers and the team 'to make sure they have the same idea of what they're looking for' (HIC review facility).This was particularly valuable if 'a review question from a policy maker [was] quite broad… [and did not] make a good systematic review question' (HIC review facility).
Successful meetings have been more structured [with] an agenda… for eliciting ideas… about conceptual definitions of terms in the question, [the] scope and geographical location of funders' priorities, to make the review manageable.(HIC review facility)

Communications template addressing engagement
Systematic review teams have been expected to provide evidence of engagement, for instance, through the peer review process (uMIC reviewer).Indeed, some templates for protocols and final reports, provided by review facilities, required review teams to describe the relevant policy background, how they engaged with policy makers, what they learnt from this engagement and how it influenced their review, or to have a policy influence plan (HIC Review Facility).
Systematic review teams were commonly required to prepare a summary that was accessible to busy policy makers.Common formats were variations on the 1-3-25 format: one page of take-home messages, a three-page executive summary that summarised the full report, and a 25-page report (Lavis et al, 2009), although some teams were better than others at producing a policy-friendly report (review facility).
In contrast, evidence-based policy briefs focused on a priority issue suggested by policy makers, and include a summary of the best available evidence, often from systematic reviews such as Cochrane reviews together with local evidence to address the issue of interest.Examples of policy briefs that have been developed include: task shifting to optimise the roles of health workers to improve the delivery of maternal and child healthcare; strengthening the health system for mental health; strategies to reduce maternal mortality. 9 For policy briefs to be useable, readers needed to understand the context from which the evidence came (local policy maker uMIC) and this was only possible if the original reviews included studies that took context into account.Evidence-Informed Policy Network (EVIPNet), which provided policy briefs that elucidated the problem and considered a few policy options and implementation considerations, was held in high regard by policy makers (two uMIC policy makers).These materials formed the basis of deliberative dialogue between key national actors to capture the tacit knowledge, views and experiences on the policy issue. 10

Establishing structures conducive to spanning worlds of policy and research
Engagement between researchers and policy makers was supported by various structures in different circumstances.

Knowledge brokers
The policy-research boundary was crossed by a knowledge-broking service in Australia which developed questions important to policy clients and answerable by academics (Campbell et al, 2011).The clients completed a structured questionnaire to clarify their information needs and provided the focus of discussion in a knowledgebrokering session; from that discussion the knowledge broker drafted the specification for the systematic review to be commissioned.Success depended largely on the characteristics of the knowledge broker who: understood both research and the policy makers' environment, possibly having worked in a government department or agency; brought credibility, respect and a level of humility that made people feel comfortable; and was at ease in an intermediate space and able to liaise efficiently with senior, knowledgeable people (HIC review facility).

Government departments and agencies
These provided their own structures for spanning research-policy boundaries.They came in the form of internal knowledge-broking teams, or the technical arms of ministries which commissioned systematic reviews (such as the Evidence into Action team at DFID) or prepared health technology assessments using systematic reviews (Ministry of Health, in Argentina), or established knowledge translation platforms (such as the National Institute for Health, in Peru).Close alignment of reviewing capacity to government departments facilitated reviews of politically sensitive questions.For instance, the Catholic Church's concern that the 'morning after' contraceptive pill may constitute abortion led to Peru's National Institute for Health conducting a systematic review that found the mechanisms of action focused on ovulation (Mezones-Holguín et al, 2011).A requirement for national programmes to be informed by systematic review evidence provided stability for review teams (uMIC policy maker).At a local level, embedding a health impact assessment unit in the policy directorate, with technical expertise drawn from public health and health systems specialists in universities, strengthened the link between research and policy (uMIC local policy maker).

Guidance development panels
The development of evidence-informed guidelines provided an opportunity for policy decisions to drive the preparation of systematic reviews.This was common for clinical guidelines, but has also been successful for health systems guidelines, although the type of evidence employed has varied.For instance, a systematic review of observational studies, complemented by experimental studies from high income countries, was 'extremely valuable' (uMIC reviewer) in identifying moderate evidence supporting targeted admission policies to enrol students with a rural background in education programmes for various health disciplines, in order to increase the likelihood of graduates subsequently choosing to practice in rural areas (WHO, 2010).The quality of evidence supporting another recommendation in the same guideline was considered low because there was no 'direct evidence on improved retention' from public recognition of rural health workers at events or with awards.However, with the 'supportive evidence from a systematic review of qualitative studies [which showed] that recognition [was] one of the main motivating factors for health workers' (WHO, 2010), the panel's strong recommendation was based on the potential impact on retention of health workers, the extremely low likelihood of adverse effects, and the very low cost of the intervention (uMIC reviewer).When preparing WHO guidelines for the most appropriate allocation of tasks between health workers delivering maternal and newborn health interventions, the guideline development panel prompted the development of methods for health systems systematic reviews (WHO, 2012).The panel, with membership representing various cadres, provided moral support and practical resources for the systematic review teams, and helped link the teams to other funding sources (HIC reviewer).Evidence was drawn from experimental and quasi-experimental studies to answer questions about the effects of delivering interventions by particular cadres of health workers; from qualitative studies addressing factors affecting the implementation of programmes to optimise the tasks and roles of cadres; and from country case studies of factors affecting the implementation of large-scale programmes for scaling up human resources (WHO, 2012).Policy input came through the guideline panel and through extensive consultation.Policy maker groups contributed to the scoping of the guidelines, and therefore the systematic reviews, and were later consulted on the review findings, especially the country case studies, when they helped make sense of the data and filled the gaps.The Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) framework, 11 under development at the time, was used to present information to the panel about each question because it could accommodate evidence about effects of intervention, acceptability and feasibility.This work prompted the development of a tool for assessing how much certainty or confidence to place in findings from qualitative evidence syntheses (Gülmezoglu et al, 2013).

International networks
All three national policy makers interviewed stressed the importance of international networks to support the use of evidence in lMICs.Where a strong evidence culture was yet to develop, technical and political support was available through international networks such as EVIPNet, National Institute for Health and Care Excellence (NICE ) International and the National Academy of Medicine (uMIC policy maker).EVIPNet was valued for its practical and moral support; its evidence-informed policy tools and the raised significance within a country that comes with membership of an international network (uMIC policy maker).It also provided a valuable role in identifying health systems reviews (rather than clinical practice reviews) that could be used for training (uMIC policy maker).

Review teams, advisory groups and review facilities
Formal engagement benefited from review teams, their advisory group and review facilities being structured to facilitate working across boundaries.Between them they needed good understanding and direct experience of the issues being addressed, review methods experience, good communicators who were willing to listen, trust between the members who brought different expertise (uMIC reviewer), and good science writers to ensure reports and briefs were engaging, jargon-free and conceptually clear (review facility, international HIC policy maker).

Standardising procedures for spanning the worlds of policy and research
Procedures for policy-relevant reviews have been standardised to varying extents in the handbooks and teaching materials of review facilities, although little is specifically for health systems. 12 13Available guidance addressed policy relevance and complexity in three different ways: by specifying the involvement of stakeholders in the production of systematic reviews; by explicit use of theory, often in discussion with stakeholders; and by expanding the range of study designs included in a review, whether broader experimental study designs for assessing effects, or a broader range of study designs to explore the theory or change.Each of these is considered below.

Guidance for stakeholder involvement
The methodological standards for the conduct of Cochrane Intervention Reviews 14 required authors to 'ensure that the review question and particularly the outcomes of interest, address issues that are important to stakeholders such as consumers, health professionals and policy makers'.A similar requirement appeared in the draft parallel standards of the Campbell Collaboration.The purpose of stakeholder involvement was clear for health systems reviews.
Because of the multidisciplinary nature of the Health Policy and Systems Research (HPSR) field, reviews will be more relevant and valid if they are informed by people with a range of experiences, in terms of both the topic and the methodology.Choosing which interventions, settings / populations, and outcomes to include in a review relies on knowledge of current policy, practice and the views of the people targeted by the interventions.Therefore, reviewers should consider the incorporation of the views from different stakeholders in a systematic way. 15  Additional roles for stakeholders included sharing decisions about the scope of the review, key concepts, search strategies, interpretation of the emerging findings and dissemination of the report (Rees and Oliver, 2012).A common model for involving stakeholders was through Advisory Groups or Review Panels that met face-toface or virtually and included expertise about the issue being addressed and the review methods (Health Policy and Systems Research Unit, 2011;Collaboration for Environmental Evidence, 2013;Joanna Briggs Institute, 2011;Rees and Oliver, 2012).The Collaboration for Environmental Evidence guidance noted that involving many people at an early stage may be particularly critical when the findings are likely to be contested (Fazey et al, 2004).However, stakeholder input needed to be carefully managed to avoid the question becoming too broad, complex or just impossible to answer (Stewart and Liabo, 2012).Review teams faced decisions about: which judgements they were willing and able to share with other stakeholders; which stakeholders could help make these judgements; who in the review team had the skills to consult or collaborate with stakeholders; what advance information would help stakeholders prepare for discussing the review; how much time was available for thought and discussion; what costs would be accrued; and how stakeholders' contributions would be acknowledged in the report (Rees and Oliver, 2012).
Where reviews were required urgently, rapid engagement has been facilitated by knowledge brokers working with standardised procedures and strict timetables (Campbell et al, 2011).The Sax Institute's commissioning tool (Campbell and Rubin, undated) encouraged policy makers to act as 'intelligent customers' (Cabinet Office, 1999) of evidence by considering and articulating: the focus and justification of the review; the timeframe and funds available to conduct the review; the breadth of evidence to be considered and the depth of analysis required; and the format of the final product.

Guidance about use of theory
Guidance for maximising the relevance of systematic reviews to policy often recommended inspecting programme theory or logic models with a broader range of study designs than was used to address effectiveness questions alone (see Figure 2).Some templates for protocols and reviews included a background section on 'how the intervention might work' while other guidance went further and recommended using an explicit theory of change, collecting data on context, implementation and outcomes along the causal chain (Waddington, 2012;Snilstveit, 2012).Some review methods have been developed specifically as theory-driven, qualitative and mixed-method approaches, to expand the knowledge base in policy-relevant areas by explaining the success, failure or mixed fortunes of complex interventions (Pawson et al, 2005).

Guidance about study designs and choice of review methods
Addressing different types of questions or contexts had implications for the choice of appropriate study designs to be included in a review.Including a broad range of experimental and quasi-experimental study designs was recommended for a number of reasons.When randomised controlled trials were not available to address questions about the effects of health system interventions and implementation strategies, a broader range of quasi-experimental studies were recommended. 16Qualitative study designs were recommended for investigating acceptability, feasibility, variability and complexity of interventions, their contexts, and explanations of their effects.How this was done was linked to use of theory in reviews.
Despite all the guidance available, when embarking on literature that had not been reviewed before, the most productive approach was not always apparent in advance.
We couldn't have followed a template when entering a completely new area.(uMIC reviewer) Attempts to match review methods to the available literature were problematic if there were too few or too many studies available.Some review facilities supported the publication of 'empty reviews' (Yaffe et al, 2012) while others adapted their reviewing methods to draw on the literature available.Decision makers wanted more than pure evidence of effectiveness (uMIC reviewer).
[A] Social Cohesion [systematic review], for example, does a neat analysis of how intervention characteristics may affect project outcomes, identifying how 'interventions have not been carried out in accordance with the theory'.This qualitative conclusion is both policy-relevant and more important, than the finding that the statistical evidence is weak and more studies are needed.(Morton et al, 2012) Policy makers could see academics' lack of consensus over research methods, for both primary research and systematic reviews.
From what I understand from the journal club and what I've been exposed to, in the research reality, there are conflicting ideologies around the primary 3ie: a tool kit (Waddington et al, 2012)

Broad range of questions and study designs
Joanna Briggs Institute Reviewers' Manual4 3ie effectiveness-plus reviews (Waddington et al, 2012) The RAMESES (Realist And Metanarrative Evidence Syntheses: Evolving Standards) project5 Collaboration for Environmental Evidence, 2013, Guidelines for systematic review and evidence synthesis in environmental management, version 4.2, environmental evidence6 Introduction to systematic reviews (Gough et al, 2012) research and the methods for research… and they are not a unified group, they are fighting one another, and it's all about the resources, and the ones with the paradigm that we want the answers from, get the least resources.
(uMIC local policy maker) Reviewers recognised such disagreements amongst themselves too.Some would have liked review facilities to be more 'flexible', encouraging more 'intellectual engagement' of review authors rather than demanding particular methods, and were concerned about setting required procedures and standards while methods are still developing (uMIC reviewer).Institutional support was needed for two groups of people: topic experts who really struggled to get their heads around the methodology; and methods experts who, working in unfamiliar areas, risked oversimplifying conceptual issues (uMIC reviewer).It was important to get the balance right between advising people and letting people find their own processes.(uMIC reviewer) Guidance for rapid reviews was key for government 17 and intermediary agencies (Campbell et al, 2011) addressing urgent problems.Rapid response reviews that searched three databases, applied no more than three key appraisal criteria, and described the studies but without a statistical meta-analysis were not methodologically as pure [as Cochrane] but [didn't] take 1-2 years, because then they're not politically relevant.(uMIC policy maker) Making an impact: products, decisions and people Having explored the diversity of approaches to systematic reviews, the differing motivations and approaches to engagement, and the supportive structures and formal procedures, it was possible to see within this rich landscape different institutional mechanisms clustering within four models for achieving policy-relevant evidence products and subsequent decisions (Figure 3).These models were distinguished largely in terms of their starting point and their purpose.Reviews could start with or without wide agreement about their key concepts (see the two rows in Figure 3), and with the purpose of offering 'public goods' for multiple audiences (as, for instance, Cochrane or Campbell reviews) or informing policy decisions with a specific jurisdiction and timescale (see the two columns in Figure 3).

Model one
Where there was good consensus about key concepts and definitions when embarking on a review addressing important, common problems, the essential institutional mechanism for producing policy-relevant systematic reviews was a comprehensive knowledge management system.This model requires harnessing the motivation of systematic reviewers to produce 'public goods' and make these readily accessible to policy makers.This model was enhanced where there were widely agreed priorities for systematic reviews (Nasser et al, 2013), standardised sets of outcomes (core outcome sets), 18 and clear typologies of interventions 19 for focusing the work, and by databases such as Health Systems Evidence and The Cochrane Library for making reviews widely available.Producing such reviews required investment in lengthy technical processes but the benefit was rigorous, credible products.The Cochrane EPOC review group provided a well-developed example of this model.

Model two
Fast response reviews addressing politically-driven priorities were possible where key concepts were clear and reviewers were embedded within, or had stable contracts with, a policy organisation with a specified jurisdiction.The result was health technology assessments or rapid reviews to inform decisions within specified contexts.This approach was endorsed by the authors of a survey of current methods and practice in health technology assessment which found an array of approaches to accelerating production (Watt et al, 2008).Where consensus about key concepts and definitions was limited, greater emphasis on engagement was required to focus the review and make sense of the findings.Essential institutional mechanisms were knowledge mediators, adequate timescales, training and support for constructive iterative engagement in order to explicate theory and make sense of the findings.Production of the WHO task-shifting guidelines followed this pathway.Keys to success were the guideline group, convened by WHO, the wide consultation for mediating stakeholder input, and the extensive discussions and considerable methodological support from leaders in the field in the absence of established review procedures.Many systematic reviews of international development more broadly have taken a similar path with mediation being provided by review teams, review facilities or knowledge brokers, and reviews taking longer to complete than anticipated.Nevertheless, investment in collective thinking and debate has paid off in terms of reviews that cut across academic disciplines to produce evidence that was either generalisable or provided enough detail about study contexts for readers to judge the wisdom of using the findings for their own concerns.

Model four
Reviews being undertaken in similar circumstances, but where the need for policyrelevant evidence is urgent, have been achieved by 'on call' review teams capable of mediating their own policy input.This capability came from their long-term experience of being hosted by or funded by government agencies, or having careers that have spanned the research-policy boundary.Alternatively, knowledge brokers have liaised between review teams and policy teams, as was done by the Sax Institute who liaised between many policy customers and academics.Either way, systematic reviewers have had opportunities to influence policy, but with the result of systematic reviews possibly being relevant only to those policy makers directly involved.

Mixed models
These models are not mutually exclusive.Model one, which is relatively slow, has been accelerated by providing additional organisational support, or by maintaining living reviews (Elliott et al, 2014).Fast response reviews (Khangura et al, 2012) in Models two and three, and reviews requiring extensive collective conceptualisation (Model four), have drawn on resources developed through Model one, and adapted them to their own ends.For instance, the National Institute of Health in Peru, when investigating the mechanisms of action for emergency contraception in order to identify a product that would not be challenged by objections to abortion, particularly held by the Catholic Church (Model two), drew on existing systematic reviews (Model one) (Mezones-Holguín et al, 2011).Conversely, reviews which were delivered to meet an urgent deadline have been developed further over time to produce more rigorous products akin to those in Model one.
Parallel to the production of policy-relevant systematic reviews was the development of evidence-literate policy makers and policy-sensitive reviewers.Where policy makers or systematic reviewers were exposed to each other's worlds, either through career changes, secondments or dual appointments, or through extensive iterative discussions through the course of producing systematic reviews, they gradually became more attuned to each other's ways of seeing.This was evident from interviews with public health scientists working in policy organisations, and from interviews with knowledge brokers or systematic reviewers with extensive experience of policy-relevant reviews.

Summary of findings
From listening to policy makers working in different contexts, systematic reviewers with varying experiences (including some in facilities that registered and published reviews), we have identified four models for choosing which institutional mechanisms to assemble to support the conduct of a policy-relevant systematic review on complex health systems questions.This choice depends primarily upon the clarity and consensus of key issues underpinning the question, and whether the review is a bespoke product or for widespread use, and results in one of a range of systematic review products.
Producing policy-relevant systematic reviews requires working sufficiently close to policy to match systematic review products appropriately to the diverse circumstances and needs of policy makers.Supporting the production of these various products can be achieved with institutional mechanisms that harness organisational and individual motivations, emphasise engagement between policy and research, and embed efforts in conducive structures and with formalised procedures.The result can be systematic reviews that are accessible, quickly achieved or cut across academic disciplines to match policy interests, but rarely all three at the same time.

Strengths and limitations of the study
The rigour of this study derives from purposively selecting for interview key organisational players who, between them, provided multiple perspectives on systems issues.This was a small study of a large system.Nevertheless, as the study progressed, successive interviews with systematic reviewers (6) and review facilities (4) revealed fewer new themes.We are less confident of having identified all relevant themes from the perspective of policy makers (7) and would anticipate identifying more from both policy makers and systematic reviewers if the focus had not been limited to health systems systematic reviews.

The wider literature
This study offers guidance on choosing between approaches to systematically reviewing health systems research depending on the starting point and the product required.Taking into account the speed of production, which this study recognised as a key distinction between different types of reviews, requires a trade-off between the breadth and depth of a review and the resources invested (Gough and Thomas, 2012).However, it is not simply a workload issue.Ironically, policy makers' typically broad questions about complex issues challenge urgent policy deadlines; and defining key concepts with stakeholders takes time (Thomas et al, 2013).Knowledge-brokering skills to facilitate policy makers and researchers working together appear to mitigate some of these problems.This finding is complemented by a systematic review which found that interpersonal relationships between research and policy are the most often mentioned facilitators of the use of evidence (Oliver et al, 2014).This study provides some answers to questions raised by Lavis et al (2005).The distinct models address the different challenges of contributing to a cumulative global stock of systematic reviews and providing context specific evidence.Early engagement with managers and policy makers can focus on improving clarity and consensus of definitions maximises relevance of systematic reviews.Such engagement needs to be supported by stakeholders or independent brokers who are capable of navigating the different priorities of research and policy while keeping a review manageable.
Interviewees in this study, in portraying policy and research as two worlds, echoed earlier experiences captured by the 'two-communities' theory (Caplan, 1979).The different mechanisms and models for bridging these two communities with systematic reviews concur with Caplan's distinction between the different community interactions required depending on the type of research or policy decisions.However, by being embedded in overlapping worlds rather than distinct communities and focusing on the functions within an overarching knowledge system, the mechanisms and models we identified suit better Wingens' (1990) reformulation of the 'twocommunities' metaphor into a systems theory that explains knowledge creation, diffusion and utilisation.

Implications for practice
The significance of this study is in its implications for generating and using systematic reviews.Evidence suggests that the most appropriate choice of institutional mechanisms to support a review depends on the degree to which the key concepts driving the question are clear and widely agreed in advance, and whether systematic reviews are being produced for a specific decision or as a public good for multiple users.
Questions to ask when choosing between approaches to ensure a review is fit for purpose include: 1. How common is the issue the review will address?Common issues benefit from reviews that are publicly available, sufficiently rigorous to provide conclusions convincing to stakeholders not immediately involved, and kept up-to-date.2. How clear and widely agreed are the underlying key concepts?Concepts can be clarified by stakeholder dialogue at the beginning and as the review progresses.3. Is the purpose of the review to test a clear hypothesis, to clarify key concepts and generate theory, or to explore existing theory?Respective approaches include aggregating findings from similar studies, configuring findings from dissimilar but related studies, or mixing these approaches within frameworks that make sense to stakeholders.4. How urgent is the problem?Reviews can be accelerated: with the help of knowledge brokers for clarifying key concepts and questions in advance; by drawing on existing systematic reviews; by more focused, less rigorous methods; or with the help of advances in information technology.Rapid reviews to inform a small group of decision-makers can be developed subsequently to produce more rigorous and widely-relevant systematic reviews.
Answers to these questions will determine which approaches to reviewing will meet policy needs, and whether a review would benefit from particular skill sets, types of engagement, structures or procedures.Aspiring to policy-relevant reviews requires investment in working relationships between the policy and research worlds.This can be through supporting relationships with a knowledge broker, benefitting from participating individuals familiar with both worlds, or giving time to developing relationships and interactive skills slowly over the course of a review or series of reviews.

Further research
The next step is to test the validity and utility of these four distinguishable and interrelated models.The immediate question is whether sharing this understanding with the producers and potential users of systematic reviews will help them assemble the appropriate institutional support mechanisms to suit each planned systematic review.

Figure 1 :
Figure 1: Analytical framework for policy relevant systematic reviews Policy makers are less concerned about the methods compared to the question and the answer.(uMIC reviewer) We just want to know, generally what's the background, what the objectives are, and what did you do in a nutshell and what were the key results.(uMIC policy maker)

Figure
Figure 2: Review guidance: theory, questions and study designs Theoretical framing of systematic review Population, intervention, Comparison, O u t c o m e s ( P I CO ) / Po p u l a t i o n , Phenomenon of Interest, Context (PICo)

Figure 3 :
Figure 3: Models for achieving policy-relevant systematic reviews

Table 1 : Roles and backgrounds of interviewees
1 Staff of review facilities were also experienced systematic reviewers 2 Review funders also held other roles3The Sax Institute facilitates the production of rapid reviews4The novice reviewer had attended training and was engaged in conducting systematic reviews, but had not completed a published review Sandra Oliver and Kelly Dickson 238